Object identification method and device
By preprocessing, feature extraction and fusion operations on multimodal data in biometric technology, the accuracy problem of object recognition in complex environments is solved, and more efficient and accurate recognition results are achieved.
Patent Information
- Application Number
- CN202411740664.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-21
AI Technical Summary
Existing biometric technologies are difficult to achieve high-precision object recognition in complex environments, mainly due to the redundancy and complexity of multimodal information.
An object recognition method is proposed, by collecting multimodal original data (image and audio data), performing preprocessing, feature extraction and fusion operations, and generating multimodal fusion data to determine the object recognition results. The specific steps include: preprocessing to generate multimodal standard data, extracting multimodal feature data, reducing redundant information through fusion rules, and finally determining the identification result.
By reducing the redundant information of multimodal data, the accuracy and robustness of object recognition are improved, and are suitable for object recognition tasks in complex environments.
Smart Images
Figure CN119672818A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of electronic technology, and in particular to an object recognition method and device. Background Art
[0002] Thanks to the rapid progress of deep learning, biometric technology has been widely used in many fields such as smart homes, animal husbandry, security monitoring and customer service, becoming one of the core research areas to promote the development of artificial intelligence.
[0003] Nevertheless, biometrics face many challenges in practical applications. Current recognition technologies based on multimodal information usually simply combine images and audio, and these features often contain a lot of redundant information after fusion, which makes it difficult to achieve high-precision object recognition in complex environments. Therefore, for the accuracy and robustness of object recognition, more advanced technologies need to be developed to deal with challenges in these complex scenarios. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention proposes an object recognition method, comprising:
[0005] Collecting multimodal raw data of the identified object, wherein the multimodal raw data includes at least image data and audio data;
[0006] Preprocessing the multimodal raw data according to a first preset rule to generate multimodal standard data;
[0007] Extracting features from the multimodal standard data according to a second preset rule to generate multimodal feature data;
[0008] Performing a fusion operation on the multimodal feature data according to a third preset rule to generate multimodal fusion data;
[0009] An object recognition result is determined according to the multimodal fusion data.
[0010] In one embodiment, the preprocessing of the multimodal raw data according to the first preset rule to generate multimodal standard data includes:
[0011] Adjust the frame rate of image data;
[0012] Adjust the sampling rate of audio data;
[0013] performing denoising and normalization operations on the adjusted image data and audio data;
[0014] At least performing serialization processing on the image data and audio data subjected to denoising and normalization operations to generate time series data;
[0015] Multimodal standard data is generated based on at least the time series data.
[0016] In one embodiment, the multimodal standard data includes audio standard data and image standard data, and feature extraction is performed on the multimodal standard data according to a second preset rule to generate multimodal feature data, including:
[0017] Extracting audio features according to audio standard data, wherein the audio features include acoustic features and prosodic features;
[0018] Extracting text features according to audio standard data, wherein the text features include word-level features and sentence-level features;
[0019] Based on the image processing algorithm, image features are extracted according to the video standard data, wherein the image features include key part features, and the key part features include head features, body features or individual features;
[0020] Multimodal feature data is generated according to the audio features, text features and image features.
[0021] In one embodiment, the extracting of acoustic features and prosodic features according to the audio standard data based on the faulty device includes:
[0022] Extract acoustic and prosodic features based on audio standard data; including:
[0023] Get the audio spectrum through Fourier transform;
[0024]
[0025] Wherein, w(n) is the window function, and x(n) is the input signal of the audio standard data;
[0026] Through Mel filter bank, logarithmic compression and DCT, the MFCC coefficients are finally obtained as acoustic features;
[0027]
[0028] Among them, H m (k) is the mth triangular bandpass filter;
[0029] Prosodic features include fundamental frequency and energy characteristics
[0030] The extracting of text features according to the audio standard data, wherein the text features include word-level features and sentence-level features, includes:
[0031] At the text word level, we combine word embedding technology and TF-IDF weights to obtain text features:
[0032] w=Embedding(word)
[0033]
[0034] At the text sentence level, an attention mechanism is used to capture text features:
[0035]
[0036] In one embodiment, the above-mentioned fusing operation on the multimodal feature data according to the third preset rule to generate multimodal fusion data includes:
[0037] Eliminate redundant information in multimodal feature data and reduce dimensional information;
[0038] Establish fusion rules for associated modal features and synthesize decision results; the fusion rules include time alignment fusion rules for image features and audio features, and facial information matching rules for image features and text features.
[0039] In one embodiment, the temporal alignment and fusion rule of the image features and the audio features mentioned above includes:
[0040] F dynamic (t) = LSTM([F audio (t),F visual (t)]);
[0041] The facial information matching rules of the image features and text features include:
[0042] F combined (t) = γF video (t)+(1-γ)F text (t)
[0043] Among them, γ is the fusion hyperparameter.
[0044] The present invention also provides an object recognition device, comprising:
[0045] An acquisition module, used for acquiring multimodal raw data of the identified object, wherein the multimodal raw data includes at least image data and audio data;
[0046] A first generating module, configured to pre-process the multimodal raw data according to a first preset rule to generate multimodal standard data;
[0047] A second generating module, used for extracting features from the multimodal standard data according to a second preset rule to generate multimodal feature data;
[0048] A third generating module, used for performing a fusion operation on the multimodal feature data according to a third preset rule to generate multimodal fusion data;
[0049] A determination module is used to determine an object recognition result according to the multimodal fusion data.
[0050] The present invention also provides an object recognition system, the object recognition system comprising: an object recognition device;
[0051] The object recognition device is determined by an object recognition method as described in the above claims.
[0052] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the object recognition method described above when executing the computer program.
[0053] The present invention also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the object recognition method described above is implemented.
[0054] The embodiment of the present invention generates multimodal standard data by collecting multimodal raw data and performing preprocessing. At the same time, the multimodal standard data is feature extracted according to the second preset rule to generate multimodal feature data; the multimodal feature data is used to replace the multimodal raw data, which greatly reduces redundant information and is also conducive to dealing with complex scenarios in practical applications. Then, the embodiment of the present invention performs a fusion operation on the multimodal feature data according to the third preset rule to generate multimodal fusion data; and determines the object recognition result according to the multimodal fusion data. The density of effective information in the multimodal fusion data is improved, the description of the object characteristics is enriched, it is conducive to quickly determining the object characteristics, and the accuracy and efficiency of recognition are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0056] Figure 1 is a flow chart of an object recognition method according to a first embodiment of the present invention;
[0057] Figure 2 is a detailed flow chart of S12 of the first embodiment of the present invention;
[0058] Figure 3is a detailed flow chart of S13 of the first embodiment of the present invention;
[0059] Figure 4 is a detailed flow chart of S14 of the first embodiment of the present invention;
[0060] Figure 5 is a structural block diagram of an object recognition device according to a second embodiment of the present invention;
[0061] Figure 6 This is a schematic diagram of the internal structure of a computer according to another embodiment of the present invention. DETAILED DESCRIPTION
[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and the well-known modules, units and their connections, links, communications or operations are not shown or described in detail. In addition, the described features, architectures or functions can be combined in any way in one or more embodiments. It should be understood by those skilled in the art that the various embodiments described below are only for illustration and not for limiting the scope of protection of the present invention. It can also be easily understood that the modules or units or processing methods in the various embodiments described herein and shown in the drawings can be combined and designed according to various different configurations. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0063] First embodiment:
[0064] Please refer to Figures 1 to 4 As shown, an embodiment of the present invention discloses an object recognition method, including:
[0065] S11, collecting multimodal original data of the identification object, wherein the multimodal original data at least includes image data and audio data.
[0066] In this embodiment, the image data of the identified object can be collected by a camera, and the audio data of the identified object can be collected by a microphone. The camera includes but is not limited to an ordinary optical camera, an infrared camera, or a 3D camera. The identified object usually includes a person or an animal, and can also include an object, etc., to expand the application scenarios of the embodiments of the present invention.
[0067] As a preferred solution but not a limitation, in some other applications, the multimodal raw data also includes 3D contour data and weight data of the object, wherein the 3D contour data of the object can be obtained by a laser radar, which continuously obtains the real-time contour of the scanned object and calculates the 3D contour data of the object through point cloud data. The weight data can be obtained by a weighing module located at a fixed position. In this preferred solution, by obtaining the 3D contour data and weight data of the object, not only the accuracy of object recognition can be improved, but also the efficiency of object recognition can be improved.
[0068] S12: preprocessing the multimodal original data according to a first preset rule to generate multimodal standard data.
[0069] Compared with the traditional multi-modal object recognition that simply combines images and audio, the embodiment of the present invention also provides a process for establishing multi-modal standard data based on time series. As a specific solution but not a limitation, please refer to Figure 2 As shown, this step S12 also includes S121-S125, which is used to perform a standardization operation on the multimodal raw data, wherein:
[0070] S121, adjusting the frame rate of the image data.
[0071] S122, adjusting the sampling rate of the audio data.
[0072] S123, performing denoising and normalization operations on the adjusted image data and audio data.
[0073] S124, performing serialization processing on at least the image data and audio data subjected to denoising and normalization operations to generate time series data.
[0074] S125, generating multimodal standard data at least based on the time series data.
[0075] In the above steps S121-S125, the frame rate of the image data and the sampling rate of the audio data are adjusted accordingly, and operations including but not limited to denoising and normalization are performed after the adjustment. Preferably, the video frame rate is controlled within the range of 25-30fps, which can ensure the continuity and integrity of the visual information and effectively reduce the computational burden. At the same time, the frame rate adjustment also needs to ensure good synchronization with the audio sampling rate, which is crucial for subsequent feature fusion.
[0076] It should be noted that, in this embodiment, S123 may also include processes such as resizing, color space transformation, data enhancement and edge detection for image data, and may also include operations such as sampling rate conversion, windowing, silent segment removal and framing for audio data. S124 generates time series data by time aligning and synchronizing the image data and audio data processed above, and performing serialization processing. The above steps S121-S125 are conducive to reducing redundant data, reducing computational complexity, and improving the efficiency and accuracy of subsequent feature extraction, while also optimizing the use of system resources and improving processing speed.
[0077] It should be noted that, for the object 3D contour data in the preferred solution of S11, S123 also includes operations such as coordinate axis alignment and contour segmentation. After the 3D contour data is preprocessed as above, it is also used as time series data in S124 to generate multimodal standard data.
[0078] S13, extracting features from the multimodal standard data according to a second preset rule to generate multimodal feature data.
[0079] This step is used to extract features from the multimodal standard data to further reduce the data size of the multimodal data. In some schemes, the feature extraction process can extract the fused multimodal standard data as a whole based on the synchronous timing. As a specific scheme rather than a limitation, in this embodiment, please refer to Figure 3 As shown, this step S13 also includes S131-S134, which are used to extract multi-modal standard data respectively, wherein:
[0080] S131, extracting audio features according to audio standard data, where the audio features include acoustic features and prosodic features.
[0081] Get the audio spectrum through Fourier transform;
[0082]
[0083] Wherein, w(n) is the window function, and x(n) is the input signal of the audio standard data;
[0084] Through Mel filter bank, logarithmic compression and DCT, the MFCC coefficients are finally obtained as acoustic features;
[0085]
[0086] Among them, H m (k) is the mth triangular bandpass filter;
[0087] Prosodic features include fundamental frequency and energy characteristics
[0088] S132, extracting text features according to the audio standard data, wherein the text features include word-level features and sentence-level features.
[0089] At the text word level, we combine word embedding technology and TF-IDF weights to obtain text features:
[0090] w=Embedding(word)
[0091]
[0092] At the text sentence level, an attention mechanism is used to capture text features:
[0093]
[0094] S133, based on an image processing algorithm, extracting image features according to video standard data, wherein the image features include key part features, and the key part features include head features, body features or individual features.
[0095] In this step S133, taking the case where the object to be identified is a human body and the application scenario is speaker identification as an example, the head features include facial contour features, expression features, facial features and lip movement features; the body features include body shape features, hairstyle features and posture features; and the individual features include moles, scars, tattoos and other features, so as to facilitate subsequent customer service or control the working logic of smart homes. Taking the case where the object to be identified is an animal and the application scenario is livestock farming as an example, by obtaining key part features including but not limited to head features, body features and individual features, the biological health / activity status, or whether there are different types of biological invasions can be identified, providing safety monitoring auxiliary data for livestock farming.
[0096] S134, generating multimodal feature data according to the audio features, text features and image features.
[0097] The above steps S131-S134 extract the multimodal standard data separately, which is beneficial to simplify the feature extraction process and improve the feature extraction accuracy on the one hand, and is beneficial to minimize the amount of useless frame data in subsequent fusion operations, reduce redundant information, increase the density of effective information, and improve feature extraction efficiency on the other hand.
[0098] S14: performing a fusion operation on the multimodal feature data according to a third preset rule to generate multimodal fusion data.
[0099] During the fusion process of this step, feature data corresponding to time frames with acoustic features, rhythmic features, word-level features, sentence-level features, head features, body features and individual features can be preferentially selected for fusion. For feature data of time frames in which one or more of the above features are missing, they can be fused in a secondary priority manner to minimize the amount of data in useless frames, reduce redundant information, increase the density of effective information, and improve feature extraction efficiency.
[0100] Please refer to Figure 4 As shown, as a specific solution but not limitation, this step S14 also includes S141-S142, wherein:
[0101] S141, eliminating redundant information in the multimodal feature data to reduce dimensional information.
[0102] S142, establishing fusion rules of associated modal features and synthesizing decision results; the fusion rules include time alignment fusion rules of image features and audio features, and facial information matching rules of image features and text features.
[0103] The temporal alignment fusion rule of the image feature and the audio feature includes:
[0104] F dynamic (t) = LSTM([F audio (t),F visual (t)]);
[0105] The facial information matching rules of the image features and text features include:
[0106] F combined (t) = γF video (t)+(1-γ)F text (t)
[0107] Among them, γ is the fusion hyperparameter.
[0108] S15, determining an object recognition result according to the multimodal fusion data.
[0109] This step S15 mainly compares the multimodal fusion data with the pre-stored data in the database to match the object recognition result. As a preferred solution but not a limitation, this embodiment can also improve the efficiency and reliability of the object recognition result by establishing an object recognition model.
[0110] The embodiment of the present invention generates multimodal standard data by collecting multimodal raw data and performing preprocessing. At the same time, the multimodal standard data is feature extracted according to the second preset rule to generate multimodal feature data; the multimodal feature data is used to replace the multimodal raw data, which greatly reduces redundant information and is also conducive to dealing with complex scenarios in practical applications. Then, the embodiment of the present invention performs a fusion operation on the multimodal feature data according to the third preset rule to generate multimodal fusion data; and determines the object recognition result according to the multimodal fusion data. The density of effective information in the multimodal fusion data is improved, the description of the object characteristics is enriched, it is conducive to quickly determining the object characteristics, and the accuracy and efficiency of recognition are improved.
[0111] Second embodiment:
[0112] Please refer to Figure 5 As shown, the present invention also provides an object recognition device 100, comprising:
[0113] The acquisition module 110 is used to acquire multimodal raw data of the identified object, wherein the multimodal raw data includes at least image data and audio data;
[0114] A first generating module 120, configured to pre-process the multimodal raw data according to a first preset rule to generate multimodal standard data;
[0115] A second generating module 130, configured to extract features from the multimodal standard data according to a second preset rule to generate multimodal feature data;
[0116] The third generating module 140 is used to perform a fusion operation on the multimodal feature data according to a third preset rule to generate multimodal fusion data;
[0117] The determination module 150 is used to determine the object recognition result according to the multimodal fusion data.
[0118] The modules of this embodiment are the same as the corresponding steps of the above method embodiment, and will not be repeated here.
[0119] The embodiment of the present invention generates multimodal standard data by collecting multimodal raw data and performing preprocessing. At the same time, the multimodal standard data is feature extracted according to the second preset rule to generate multimodal feature data; the multimodal feature data is used to replace the multimodal raw data, which greatly reduces redundant information and is also conducive to dealing with complex scenarios in practical applications. Then, the embodiment of the present invention performs a fusion operation on the multimodal feature data according to the third preset rule to generate multimodal fusion data; and determines the object recognition result according to the multimodal fusion data. The density of effective information in the multimodal fusion data is improved, the description of the object characteristics is enriched, it is conducive to quickly determining the object characteristics, and the accuracy and efficiency of recognition are improved.
[0120] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0121] The present invention also provides a computer device, an object recognition system, the object recognition system comprising: an object recognition device;
[0122] The object recognition device is determined by an object recognition method shown in the first embodiment above.
[0123] An embodiment of the present invention further provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the object recognition method in the above embodiments is implemented.
[0124] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned object recognition methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0125] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.
[0126] Corresponding to the above-mentioned computer storage medium, in one embodiment, a computer device is also provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an object recognition method as in the above-mentioned embodiments is implemented.
[0127] The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an object recognition method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0128] The embodiment of the present invention generates multimodal standard data by collecting multimodal raw data and performing preprocessing. At the same time, the multimodal standard data is feature extracted according to the second preset rule to generate multimodal feature data; the multimodal feature data is used to replace the multimodal raw data, which greatly reduces redundant information and is also conducive to dealing with complex scenarios in practical applications. Then, the embodiment of the present invention performs a fusion operation on the multimodal feature data according to the third preset rule to generate multimodal fusion data; and determines the object recognition result according to the multimodal fusion data. The density of effective information in the multimodal fusion data is improved, the description of the object characteristics is enriched, it is conducive to quickly determining the object characteristics, and the accuracy and efficiency of recognition are improved.
[0129] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0130] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. An object recognition method, characterized in that: include: Collecting multimodal raw data of the identified object, wherein the multimodal raw data includes at least image data and audio data; Preprocessing the multimodal raw data according to a first preset rule to generate multimodal standard data; Extracting features from the multimodal standard data according to a second preset rule to generate multimodal feature data; Performing a fusion operation on the multimodal feature data according to a third preset rule to generate multimodal fusion data; An object recognition result is determined according to the multimodal fusion data.
2. The method according to claim 1, characterized in that The preprocessing of the multimodal raw data according to the first preset rule to generate multimodal standard data includes: Adjust the frame rate of image data; Adjust the sampling rate of audio data; performing denoising and normalization operations on the adjusted image data and audio data; At least performing serialization processing on the image data and audio data subjected to denoising and normalization operations to generate time series data; Multimodal standard data is generated based on at least the time series data.
3. The method according to claim 2, characterized in that The multimodal standard data includes audio standard data and image standard data. Feature extraction is performed on the multimodal standard data according to a second preset rule to generate multimodal feature data, including: Extracting audio features according to audio standard data, wherein the audio features include acoustic features and prosodic features; Extracting text features according to audio standard data, wherein the text features include word-level features and sentence-level features; Based on the image processing algorithm, image features are extracted according to the video standard data, wherein the image features include key part features, and the key part features include head features, body features or individual features; Multimodal feature data is generated according to the audio features, text features and image features.
4. The method according to claim 3, characterized in that The extracting of acoustic features and prosodic features according to the audio standard data based on the faulty device includes: Extract acoustic and prosodic features based on audio standard data; including: Get the audio spectrum through Fourier transform; Wherein, w(n) is the window function, and x(n) is the input signal of the audio standard data; Through Mel filter bank, logarithmic compression and DCT, the MFCC coefficients are finally obtained as acoustic features; Among them, H m (k) is the mth triangular bandpass filter; Prosodic features include fundamental frequency and energy characteristics The extracting of text features according to the audio standard data, wherein the text features include word-level features and sentence-level features, includes: At the text word level, we combine word embedding technology and TF-IDF weights to obtain text features: w=Embedding(word) At the text sentence level, an attention mechanism is used to capture text features:
5. The method according to claim 4, characterized in that The step of performing a fusion operation on the multimodal feature data according to the third preset rule to generate multimodal fusion data includes: Eliminate redundant information in multimodal feature data and reduce dimensional information; Establish fusion rules for associated modal features and synthesize decision results; the fusion rules include time alignment fusion rules for image features and audio features, and facial information matching rules for image features and text features.
6. The method according to claim 5, characterized in that The temporal alignment fusion rule of the image feature and the audio feature includes: F dynamic (t)=LSTM([F audio (t),F visual (t)]); The facial information matching rules of the image features and text features include: F combined (t)=γF video (t)+(1-γ)F text (t) Among them, γ is the fusion hyperparameter.
7. An object recognition device, characterized in that: include: An acquisition module, used for acquiring multimodal raw data of the identified object, wherein the multimodal raw data includes at least image data and audio data; A first generating module, configured to pre-process the multimodal raw data according to a first preset rule to generate multimodal standard data; A second generating module, used for extracting features from the multimodal standard data according to a second preset rule to generate multimodal feature data; A third generating module, used for performing a fusion operation on the multimodal feature data according to a third preset rule to generate multimodal fusion data; A determination module is used to determine an object recognition result according to the multimodal fusion data.
8. An object recognition system, characterized in that: The object recognition system comprises: an object recognition device; The object recognition device is determined by an object recognition method according to any one of claims 1-6.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the object recognition method according to any one of claims 1 to 6 is implemented.
10. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the object recognition method according to any one of claims 1 to 6 is implemented.