Road scene recognition method and system based on audio-visual fusion
Through audio-visual fusion technology, combining the local and global features of video and fusion of visual and audio information, the problem of inaccurate identification in the prior art is solved, and the accuracy and fine-grainedness of road video scene recognition is improved.
Patent Information
- Application Number
- CN202410992437.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing road scene recognition technology mainly relies on image information, making it difficult to accurately characterize the global background information and fine-grained target information in video scenes, resulting in inaccurate identification.
Using an audio-visual fusion-based method, the local and global features of the video are extracted, and visual features are multi-modal fusion with audio features (such as MFCC), and road video scene recognition is used using a multi-layer audio-visual fusion framework.
It improves the accuracy of road video scene recognition, can more comprehensively represent scene semantics, and enhances the fine-grainedness and accuracy of recognition.
Smart Images

Figure CN118965103A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of road scene recognition, and in particular relates to a road scene recognition method and system based on audio-visual fusion. Background Art
[0002] With the development of computer vision and driverless technology, road scene recognition technology has been increasingly used, with increasingly mature technical solutions and higher accuracy. At present, road scene recognition is mostly based on images and does not involve information in other modalities such as sound. In fact, video can better reflect the real situation of the road over a period of time, and with the assistance of sound, it can better reflect the real scene. There are two video scene representation methods currently used in video scene recognition: one is to use a local target set in the scene to represent the scene information, and the other is to use global video features to represent the scene information.
[0003] However, the inventors found that simply using a target set tends to ignore global background information, and simply using global video features cannot represent the targets contained in the scene in a fine-grained manner, thus resulting in inaccurate video scene representation. Summary of the invention
[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a road scene recognition method and system based on audio-visual fusion, which fully considers the combination of local target features and global scene features of road video scenes, and considers multimodal fusion of visual features and sound features. Through the framework of multi-layer audio-visual fusion, the accuracy of road video scene recognition is improved.
[0005] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:
[0006] A first aspect of the present invention provides a road scene recognition method based on audio-visual fusion.
[0007] The road scene recognition method based on audio-visual fusion includes the following steps:
[0008] Obtain road scene video and extract key frames;
[0009] Extract global features and local features of each frame in the key frame, fuse the global features and local features of each frame to obtain multi-layer key frame scene feature representation, fuse the multi-layer key frame scene feature representation to obtain the visual features of the road scene visual modality;
[0010] Separate the audio in the road scene video and extract the MFCC features of the audio as the sound features of the road scene audio mode;
[0011] The visual features and the sound features are fused to obtain fused features, and the road scenes are classified based on the fused features to obtain road scene recognition results.
[0012] A second aspect of the present invention provides a road scene recognition system based on audio-visual fusion.
[0013] The road scene recognition system based on audio-visual fusion includes:
[0014] The key frame extraction module is configured to: obtain a road scene video and extract a key frame;
[0015] The visual feature extraction module is configured to: extract global features and local features of each frame in the key frame, fuse the global features and local features of each frame to obtain a multi-layer scene feature representation of the key frame, and fuse the multi-layer scene feature representation of the key frame to obtain a visual feature of the road scene visual modality;
[0016] The sound feature extraction module is configured to: separate the audio in the road scene video and extract the MFCC features of the audio as the sound features of the road scene audio modality;
[0017] The classification module is configured to: fuse the visual features and the sound features to obtain fused features, classify the road scene based on the fused features, and obtain a road scene recognition result.
[0018] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the road scene recognition method based on audio-visual fusion as described in the first aspect of the present invention.
[0019] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the road scene recognition method based on audio-visual fusion as described in the first aspect of the present invention are implemented.
[0020] One or more of the above technical solutions have the following beneficial effects:
[0021] The present invention provides a road scene recognition method and system based on audio-visual fusion. Aiming at the current situation that images are mostly used for recognition in road scene recognition, a multi-layer audio-visual fusion road video scene classification network is proposed. The visual and sound information in the road scene video are fused through the multi-layer audio-visual fusion road video scene classification network, and the sound information is integrated into the video scene recognition. The scene semantics is learned by fusing the audio-visual two modalities. The obtained fusion features can better represent the scene semantics, thereby improving the accuracy of road video scene recognition.
[0022] In view of the limitations of global information and local information in representing scene semantics in road video scene recognition, the present invention first detects and segments the target area of the key frame in the extraction of local scene information, and extracts the features of the target area based on the segmentation result based on the VGG16 convolutional neural network, which can extract the local features of the target area more specifically, and use the representation method of integrating local features with global features to represent the scene features of the key frame, which can better and more comprehensively represent the scene semantics.
[0023] The concept of the present invention is applicable to a variety of machine vision scenarios including autonomous driving, and can also be referenced in a variety of machine vision tasks to help them complete the tasks better.
[0024] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0026] Figure 1 This is the overall architecture diagram of the road video scene classification network with multi-layer audio-visual fusion in the first embodiment.
[0027] Figure 2 This is a flow chart of the method of the first embodiment. DETAILED DESCRIPTION
[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0029] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.
[0030] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0031] Embodiment 1
[0032] This embodiment discloses a road scene recognition method based on audio-visual fusion.
[0033] As mentioned above, in order to solve the problem of inaccurate video scene representation in the prior art and improve the representation capability of road video scene features, the present invention uses a combination of local target features and global scene features to represent the visual features of the video scene. Since the video contains audio information, the audio information has a good auxiliary effect on road video scene recognition. Therefore, this embodiment fuses the road scene visual features with the sound features, classifies and recognizes the road scene based on the fused features, and constructs a road video scene recognition method based on multi-layer audio-visual fusion.
[0034] Further:
[0035] In the visual modality, the key frames of the video are used for target detection to obtain the foreground target in the video, and the visual features of the foreground target area are extracted to obtain the local scene feature representation. At the same time, the key frames are used for global scene feature extraction, and the local features are fused with the global features to obtain the scene feature representation of the visual modality;
[0036] The audio is separated from the video, the audio features are extracted, and then fused with the scene feature representation of the visual modality, namely the visual features, to form a multi-layer audio-visual fusion video scene feature representation.
[0037] Through multi-layer audio-visual fusion, semantically rich road scene feature representations can be learned to improve the accuracy of road video scene recognition.
[0038] More specifically, Figure 1 and Figure 2 As shown, the road scene recognition method based on audio-visual fusion includes the following steps:
[0039] Obtain road scene video and extract key frames;
[0040] Extract global features and local features of each frame in the key frame, fuse the global features and local features of each frame to obtain multi-layer key frame scene feature representation, fuse the multi-layer key frame scene feature representation to obtain the visual features of the road scene visual modality;
[0041] Separate the audio in the road scene video and extract the MFCC features of the audio as the sound features of the road scene audio mode;
[0042] The visual features and the sound features are fused to obtain fused features, and the road scenes are classified based on the fused features to obtain road scene recognition results.
[0043] Among them, the MFCC feature is the Mel-scale Frequency Cepstral Coefficients (MFCC for short), which is the cepstral parameter extracted in the Mel scale frequency domain. The Mel scale describes the nonlinear characteristics of the human ear frequency.
[0044] The overall network structure used in the road scene recognition method based on audio-visual fusion in this embodiment generally includes two parts: a multi-level video scene representation network and a multimodal classification network:
[0045] The multi-level video scene representation network processes data in the following ways:
[0046] S1: Extract key frames from the video by taking one frame every 5 frames, and select the first 50 key frames of each video.
[0047] S2: Extract key frame global features.
[0048] The global features are extracted in the following way: all key frames are resized to 224*224 by the resize method and input into the VGG16_Places365 network, which outputs 4096-dimensional image features as the visual modality features of each frame.
[0049] VGG16_Places365 is a mature network model in the prior art and will not be described in detail here.
[0050] S3: Extract local features of key frames.
[0051] The local features are extracted in the following way: the image size is unified to 640*640 by resizing method, and it is input into the pre-trained YoLoV5 algorithm to detect the target in the video frame, the target area is segmented by opencv, and the segmentation results of all target areas are uniformly expanded to the same size of 224*224;
[0052] The segmentation results of the target regions of uniform size are input into the VGG16 convolutional neural network to extract 4096-dimensional features of all target regions;
[0053] The features of all target areas of the same key frame are fused by adding them together to obtain the local feature representation of each key frame.
[0054] YoLoV5 is a target detection model used to identify and locate objects in videos or images. It uses a convolutional neural network (CNN) to learn the features of objects in images, and uses multi-scale prediction and network segmentation to detect and locate targets. It adopts a single-stage target detection method and regards the entire target detection process as a regression problem, directly predicting the target's bounding box and category from the image.
[0055] YoLoV5 is a mature network model in the prior art and will not be described in detail here.
[0056] S4: Fusion of keyframe global features and local features.
[0057] The global features and local features of the key frames extracted from S2 and S3 are fused by addition to obtain a multi-layer key frame scene feature representation.
[0058] At the same time, from Figure 1 and Figure 2 It can be seen that the global features and local features of the key frames extracted in S2 and S3 are performed in parallel.
[0059] S5: Temporal feature representation.
[0060] The scene feature representations of all key frames obtained in S4 are input into the Visual Transformer as tokens (minimal unit sequences). Through the encoder of the transformer, the scene feature representations of all key frames are fused to obtain the visual features of the road scene visual modality with time sequence preservation. The visual features obtained after the time sequence feature representation in S5 are Figure 2 The spatiotemporal characteristics of .
[0061] Visual Transformer is a mature network model in the prior art and will not be described in detail here.
[0062] Furthermore, the multimodal classification network comprises:
[0063] S1: Use FFmpeg to separate the audio from the video, extract the MFCC features of the audio, and use the MFCC features as the feature representation of the sound modality, that is, the sound features, recorded as Where n is the sample number and a represents the sound mode.
[0064] FFmpeg is an open source computer program that can be used to record, convert digital audio and video, and convert them into streams. It will not be described in detail here.
[0065] S2: The visual features extracted by the multi-level video scene representation network are recorded as Where n is the sample number and v represents the visual modality.
[0066] S3: The visual features and the sound features are connected to form a new feature representation, namely the fused feature. The fused feature is input into a multi-layer fully connected neural network, and the recognition result is obtained through softmax.
[0067] S4: Training process.
[0068] The loss function of the entire multimodal classification network adopts cross entropy loss. The classification network is trained by minimizing the cross entropy loss and the back propagation algorithm until the classification accuracy reaches a certain set value or the algorithm converges, ending the training process.
[0069] S5: Testing process.
[0070] The untrained test set samples are input into the overall network structure used in the trained audio-visual fusion-based road scene recognition method, and the final recognition and classification results are obtained through a multi-level video scene representation network and a multimodal classification network.
[0071] The present invention fully considers the combination of local target features and global scene features of road video scenes, and considers multimodal fusion of visual features and sound features. Through a multi-layer audio-visual fusion framework, the accuracy of road video scene recognition is improved.
[0072] Embodiment 2
[0073] This embodiment discloses a road scene recognition system based on audio-visual fusion.
[0074] The road scene recognition system based on audio-visual fusion includes:
[0075] The key frame extraction module is configured to: obtain a road scene video and extract a key frame;
[0076] The visual feature extraction module is configured to: extract global features and local features of each frame in the key frame, fuse the global features and local features of each frame to obtain a multi-layer scene feature representation of the key frame, and fuse the multi-layer scene feature representation of the key frame to obtain a visual feature of the road scene visual modality;
[0077] The sound feature extraction module is configured to: separate the audio in the road scene video and extract the MFCC features of the audio as the sound features of the road scene audio modality;
[0078] The classification module is configured to: fuse the visual features and the sound features to obtain fused features, classify the road scene based on the fused features, and obtain a road scene recognition result.
[0079] Embodiment 3
[0080] The purpose of this embodiment is to provide a computer-readable storage medium.
[0081] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the road scene recognition method based on audio-visual fusion as described in Example 1 of the present disclosure.
[0082] Embodiment 4
[0083] The purpose of this embodiment is to provide an electronic device.
[0084] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the road scene recognition method based on audio-visual fusion as described in Example 1 of the present disclosure are implemented.
[0085] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.
[0086] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.
[0087] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A road scene recognition method based on audio-visual fusion, characterized in that: The following steps are involved: Obtain road scene video and extract key frames; Extract global features and local features of each frame in the key frame, fuse the global features and local features of each frame to obtain multi-layer key frame scene feature representation, fuse the multi-layer key frame scene feature representation to obtain the visual features of the road scene visual modality; Separate the audio in the road scene video and extract the MFCC features of the audio as the sound features of the road scene audio mode; The visual features and the sound features are fused to obtain fused features, and the road scenes are classified based on the fused features to obtain road scene recognition results.
2. The road scene recognition method based on audio-visual fusion according to claim 1, characterized in that: The extracted key frames are input into the VGG16_Places365 network to extract image features and obtain the global features of each frame.
3. The road scene recognition method based on audio-visual fusion according to claim 1, characterized in that: Detect all target areas of the extracted key frames and segment the target areas; The segmentation results of all target areas are unified in size and input into the VGG16 convolutional neural network to extract the features of all target areas; The features extracted from all target areas of the same key frame are added and fused to obtain the local features of each frame.
4. The road scene recognition method based on audio-visual fusion according to claim 1, characterized in that: The scene features of multiple layers of key frames are input into the Visual Transformer as tokens. Through the encoder of the transformer, the scene features of all key frames are fused to obtain the visual features of the road scene visual modality.
5. The road scene recognition method based on audio-visual fusion as claimed in claim 1, characterized in that: The visual features and the sound features are fused by connecting to obtain fused features; The fused features are input into a multi-layer fully connected neural network, and the road scene recognition results are obtained through softmax.
6. The road scene recognition method based on audio-visual fusion as claimed in claim 3, characterized in that: The target area of the key frame is detected based on the pre-trained YoLoV5 algorithm, and the target area is segmented through OpenCV.
7. The road scene recognition method based on audio-visual fusion according to claim 1, characterized in that: Use FFmpeg to separate the audio from the road scene video.
8. The road scene recognition system based on audio-visual fusion is characterized by: include: The key frame extraction module is configured to: obtain a road scene video and extract a key frame; The visual feature extraction module is configured to: extract global features and local features of each frame in the key frame, fuse the global features and local features of each frame to obtain a multi-layer scene feature representation of the key frame, and fuse the multi-layer scene feature representation of the key frame to obtain a visual feature of the road scene visual modality; The sound feature extraction module is configured to: separate the audio in the road scene video and extract the MFCC features of the audio as the sound features of the road scene audio modality; The classification module is configured to: fuse the visual features and the sound features to obtain fused features, classify the road scene based on the fused features, and obtain a road scene recognition result.
9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the road scene recognition method based on audio-visual fusion as described in any one of claims 1 to 7 are implemented.
10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the road scene recognition method based on audio-visual fusion as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Traffic scene understanding method and device based on video streaming
CN112347933A
Scene adaptive hearing aid audio enhancement system based on neural network
CN114245280A
Video scene recognition method and system based on knowledge enhancement
CN115410130A
Small sample abnormal behavior identification method based on key area and scene depth
CN115439926A
Multi-modal speech recognition method based on visual scene, electronic equipment and medium
CN118155624A
