Object recognition method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an object recognition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring target to-be-analyzed data containing a target object; performing reasoning analysis on the target data to be analyzed according to the multi-modal reasoning large model to obtain a dynamic reasoning path of the target object; and in combination with the guidance of the dynamic reasoning path, reasoning the target to-be-analyzed data through the multi-modal reasoning large model to obtain an identification result of the target object, and generating a target explanatory description text according to the dynamic reasoning path and a preset mapping dictionary. By adopting the method, the interpretability of the object analysis result can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an object recognition method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of artificial intelligence technology, the perception and data processing capabilities of machine learning models are becoming more and more mature. By applying machine learning models, state recognition can be achieved for target objects, such as recognizing user emotions, language features, facial expressions, etc.

[0003] In traditional technology, for the recognition of psychological state, a user can perform frame-by-frame analysis and processing on a target object's facial video through a pre-trained CNN (Convolutional Neural Networks) model or a Transformer model, map the features of the input video data to a label space representing the state of the target object, and perform state mapping of the target object in the label space to obtain a state recognition result of the target object.

[0004] However, in the current traditional technology, since the CNN model or the Transformer model performs state analysis by mapping video features to label results, the dimensionality of the object analysis result is relatively single, which leads to poor explainability of the object analysis result. SUMMARY

[0005] Therefore, it is necessary to provide an object recognition method, device, computer equipment, computer readable storage medium and computer program product to solve the above technical problems.

[0006] In a first aspect, the present application provides an object recognition method, comprising:

[0007] obtaining target analysis data containing a target object;

[0008] performing inference analysis on the target analysis data according to a multi-modal inference large model to obtain a dynamic inference path of the target object;

[0009] performing inference on the target analysis data through the multi-modal inference large model under the guidance of the dynamic inference path to obtain a recognition result of the target object, and generating a target explainability description text according to the dynamic inference path and a preset mapping dictionary.

[0010] In one embodiment, the target analysis data includes key frame images and target audio segments; and the obtaining of the target analysis data containing the target object comprises:

[0011] Obtaining original to-be-analyzed data; the original to-be-analyzed data includes video data and audio data;

[0012] According to a preset sampling strategy and a preset time window, key frame extraction is performed on the video data to obtain a key frame image corresponding to each preset time window.

[0013] Based on the preset time window, the audio data is divided to obtain the target audio segment corresponding to each preset time window.

[0014] In one embodiment, according to the preset sampling strategy and the preset time window, the key frame extraction is performed on the video data to obtain the key frame image corresponding to each preset time window, including:

[0015] The feature type of the audio data is identified to determine the feature type of each time period in the audio data.

[0016] Based on the feature type, the target sampling frequency of the video data in each preset time window is determined, and the video data is sampled according to the target sampling frequency to obtain the key frame image corresponding to each preset time window.

[0017] In one embodiment, according to the preset sampling strategy and the preset time window, the key frame extraction is performed on the video data to obtain the key frame image corresponding to each preset time window, including:

[0018] According to the default sampling frequency, the video data in the original analysis data is sampled to obtain the initial key frame image of each preset time window, and the initial audio segment corresponding to the initial key frame image is determined.

[0019] Based on the initial key frame image and the initial audio segment, an initial recognition result is obtained.

[0020] According to the initial recognition result, the default sampling frequency is dynamically adjusted to obtain a target sampling frequency, and the video data is sampled according to the target sampling frequency to obtain the key frame image corresponding to each preset time window.

[0021] In one embodiment, the multi-task output head of the multi-modal reasoning large model includes a classification head and a text generation head; the target to-be-analyzed data is reasoned by the multi-modal reasoning large model according to the guidance of the dynamic reasoning path to obtain the recognition result of the target object, and a target explanatory description text is generated according to the dynamic reasoning path and a preset mapping dictionary, including:

[0022] Under the guidance of the dynamic inference path, the target to-be-analyzed data is analyzed by inference based on the classification head, and an identification result of the target object is obtained.

[0023] An initial explanatory description text corresponding to the identification result is generated according to the text generation head and the dynamic inference path.

[0024] The initial explanatory description text is mapped according to a preset mapping dictionary, and a target explanatory description text is obtained.

[0025] In one of the embodiments, if the identification result is an abnormal result, after the target explanatory description text is generated according to the dynamic inference path and the preset mapping dictionary, the method further includes:

[0026] Each preset dimension in the target explanatory description text is analyzed by inference according to the multi-modal inference large model, and an initial analysis result corresponding to each preset dimension is obtained.

[0027] Each initial analysis result is processed according to a preset weight corresponding to each preset dimension, and a tendency degree of the analysis result of the target object is obtained.

[0028] In a second aspect, the present application further provides an object identification device, comprising:

[0029] An acquisition module is configured to acquire target to-be-analyzed data containing a target object.

[0030] A first inference module is configured to analyze the target to-be-analyzed data by inference according to a multi-modal inference large model, and obtain a dynamic inference path of the target object.

[0031] A second inference module is configured to analyze the target to-be-analyzed data by inference through the multi-modal inference large model under the guidance of the dynamic inference path, obtain an identification result of the target object, and generate a target explanatory description text according to the dynamic inference path and a preset mapping dictionary.

[0032] In one of the embodiments, the target to-be-analyzed data includes key frame images and target audio segments; the acquisition module is specifically configured to acquire original to-be-analyzed data; and the original to-be-analyzed data includes video data and audio data.

[0033] The video data is extracted by key frames according to a preset sampling strategy and a preset time window, and a key frame image corresponding to each preset time window is obtained.

[0034] The audio data is divided based on the preset time window, and a target audio segment corresponding to each preset time window is obtained.

[0035] In one of the embodiments, the obtaining module is specifically configured to perform feature type identification on the audio data, and determine feature types of time periods in the audio data.

[0036] The target sampling frequency of the video data in each of the preset time windows is determined based on the feature types, and the video data is sampled according to the target sampling frequency to obtain a key frame image corresponding to each of the preset time windows.

[0037] In one of the embodiments, the obtaining module is specifically configured to sample the video data in the original analysis data according to a default sampling frequency to obtain initial key frame images of the preset time windows, and determine initial audio segments corresponding to the initial key frame images.

[0038] An initial recognition result is obtained based on the initial key frame images and the initial audio segments.

[0039] The default sampling frequency is dynamically adjusted according to the initial recognition result to obtain a target sampling frequency, and the video data is sampled according to the target sampling frequency to obtain a key frame image corresponding to each of the preset time windows.

[0040] In one of the embodiments, the multi-task output head of the multi-modal inference large model includes a classification head and a text generation head; the second inference module is specifically configured to perform inference analysis on the target analysis data based on the classification head under the guidance of the dynamic inference path to obtain a recognition result of the target object.

[0041] An initial explanatory description text corresponding to the recognition result is generated according to the text generation head and the dynamic inference path.

[0042] A target explanatory description text is obtained by mapping the initial explanatory description text according to a preset mapping dictionary.

[0043] In one of the embodiments, if the recognition result is an abnormal result, the device further includes:

[0044] A third inference module is configured to perform inference analysis on description texts corresponding to each of preset dimensions in the target explanatory description text according to the multi-modal inference large model to obtain an initial analysis result corresponding to each of the preset dimensions.

[0045] A calculation module is configured to process the initial analysis results according to preset weights corresponding to each of the preset dimensions to obtain an analysis result tendency degree of the target object.

[0046] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0047] obtaining target data to be analyzed containing a target object;

[0048] performing inference analysis on the target data to be analyzed according to a multi-modal inference large model to obtain a dynamic inference path of the target object;

[0049] performing inference on the target data to be analyzed through the multi-modal inference large model under the guidance of the dynamic inference path to obtain an identification result of the target object, and generating a target explanatory description text according to the dynamic inference path and a preset mapping dictionary.

[0050] In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the following steps:

[0051] obtaining target data to be analyzed containing a target object;

[0052] performing inference analysis on the target data to be analyzed according to a multi-modal inference large model to obtain a dynamic inference path of the target object;

[0053] performing inference on the target data to be analyzed through the multi-modal inference large model under the guidance of the dynamic inference path to obtain an identification result of the target object, and generating a target explanatory description text according to the dynamic inference path and a preset mapping dictionary.

[0054] In a fifth aspect, the present application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the following steps:

[0055] obtaining target data to be analyzed containing a target object;

[0056] performing inference analysis on the target data to be analyzed according to a multi-modal inference large model to obtain a dynamic inference path of the target object;

[0057] performing inference on the target data to be analyzed through the multi-modal inference large model under the guidance of the dynamic inference path to obtain an identification result of the target object, and generating a target explanatory description text according to the dynamic inference path and a preset mapping dictionary.

[0058] The aforementioned object recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire target data containing the target object to be analyzed; perform reasoning analysis on the target data based on a multimodal reasoning model to obtain a dynamic reasoning path for the target object; and, guided by the dynamic reasoning path, use the multimodal reasoning model to reason about the target data to obtain the recognition result of the target object, and generate a target explanatory description text based on the dynamic reasoning path and a preset mapping dictionary. Using this method, the dynamic reasoning path obtained through the multimodal reasoning model's reasoning analysis of the target data can capture dynamic reasoning paths in the current target data that reflect the characteristics of the target object. This dynamic reasoning path can not only guide the multimodal reasoning model's reasoning for the target object recognition but also serve as the basis for generating the target explanatory description text (which can be direct or indirect), making the reasoning of the recognition result strongly correlated with the target explanatory description text. Therefore, the target explanatory description text can reflect the reasoning logic of the recognition result. Furthermore, the target explanatory description text is used as a supplement to the output of the multimodal reasoning big model. The target explanatory description text enriches the dimensions of the object analysis results, thereby improving the interpretability of the object analysis results. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating an object recognition method in one embodiment;

[0061] Figure 2 This is a schematic diagram illustrating the process of sampling video data and segmenting audio data to obtain the target data to be analyzed in one embodiment.

[0062] Figure 3 This is a flowchart illustrating the process of determining the target sampling frequency based on feature type in one embodiment;

[0063] Figure 4 This is a flowchart illustrating the process of determining the target sampling frequency based on the initial identification results in one embodiment;

[0064] Figure 5 This is a flowchart illustrating the process of generating target explanatory description text based on dynamic reasoning paths in one embodiment.

[0065] Figure 6A flowchart of a process for quantifying the tendency result of the analysis result of the target object in one embodiment;

[0066] Figure 7 A flowchart of an example of an object recognition method in one embodiment;

[0067] Figure 8 A structure block diagram of an object recognition apparatus in one embodiment;

[0068] Figure 9 An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0069] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0070] It should be noted that the terms "first", "second" and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two and more than two. The term "and / or" used in the present application refers to one of the options or any combination of multiple options.

[0071] The user data, data acquisition and / or use that the embodiments of the present application can involve strictly follow the relevant national and regional laws, regulations and industry standards. The collection and acquisition of data involved in the embodiments of the present application are all previously informed to the user by active prompt or prominent identification and the like, and authorized or fully authorized by all parties. The processing, processing, retransmission and use of data involved in the embodiments of the present application are all carried out on the premise that the user or the relevant party is fully informed and authorized. In implementing the embodiments of the present application, the user or the relevant party is informed of the data or information types, use range, use scenario and the like that can be involved by appropriate means and authorized, and the specific informing and authorization means can vary according to the actual situation, and the present application is not limited in this respect. The personal information processing involved in the embodiments of the present application is all carried out on the basis of legality (such as the consent of the subject of personal information or necessity for performing a contract), and only within the prescribed or agreed range. The sensitive personal information such as biometric information, medical and health information, financial account information and precise positioning information involved in the embodiments of the present application is processed on the premise of specific purpose and sufficiency, and the separate authorization consent of the user or the relevant party is obtained. In some embodiments of the present application, if the user or the relevant party refuses to process the personal information other than the necessary information required for the basic function, the use of the basic function of the embodiments of the present application will not be affected.

[0072] In one embodiment, as shown in Figure 1 A method for object recognition is provided, and the embodiments are exemplified by applying the method to a terminal. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction of the terminal and the server. The method includes the following steps:

[0073] In step 102, target to-be-analyzed data containing a target object is acquired.

[0074] In the embodiments of the present application, the target object is a to-be-identified object, that is, to identify whether the target object has a negative emotional disorder or a negative emotional state, for example, to identify whether the target object is sad, and for example, to identify whether the target object has depression. The target to-be-analyzed data is multi-modal data containing facial images and audio data in an interview of the target object. For example, the target object can be a patient to be analyzed, and the clinical interview process of the patient is recorded in the form of audio and video recording, the facial expression features and the audio of the patient's answers to questions during the interview process are recorded to obtain original audio and video data. Then, the terminal extracts images from the video part of the original audio and video data according to a pre-set sparse sampling strategy, splices the extracted key frame images with the corresponding audio segments to obtain target to-be-analyzed data, and stores the target to-be-analyzed data to the terminal. The target to-be-analyzed data is used as input data for analysis and processing of the multi-modal inference large model. The original audio and video data is compressed and extracted, and the data volume for inference analysis of the multi-modal large model is reduced.

[0075] Therefore, when object recognition of the target object is needed, the terminal reads the to-be-analyzed data containing the target object from the storage location specified by the user, and further inputs the target to-be-analyzed data into the multi-modal inference large model.

[0076] Step 104, performing inference analysis on the target to-be-analyzed data according to the multi-modal inference large model to obtain a dynamic inference path of the target object.

[0077] In the embodiments of the present application, the multi-modal inference large model is obtained by adding a multi-task output head to a pre-trained large language model and performing light-weight fine-tuning. The multi-task output head includes a classification head, a regression head, and a text generation head. The classification head is used to determine whether the target object has a negative emotional disorder or a negative emotional state, and outputs a category label. For example, the category label can be "depression" or "non-depression". The regression head is used to predict an index of the severity of the negative emotional disorder or the negative emotional state of the target object, and outputs a continuous numerical index. The text generation head is used to generate a readable explanation for the category label analysis, and to describe the inference basis and the features used for judgment in the form of text.

[0078] For the light fine-tuning of the multi-modal large model, the positive samples are data representing individuals with negative emotional disorders or negative emotional states, for example, the positive samples can be individual data of clinically diagnosed depression patients, and for another example, the positive samples can be individual data determined to be in negative emotions such as sadness, sorrow, or anxiety; the negative samples are normal emotional control group data, for example, the negative samples can be individual data of subjects with normal emotional states. In some embodiments, the normal emotion can be an emotion other than the negative emotional state. And both the positive and negative samples contain description texts for emotional state reasoning, which represent reasoning texts for judging that the individual data has a negative emotional disorder or is in a negative emotional state, and reasoning texts for judging that the individual data is in a normal emotion, for example, the description texts can be the judgment basis written by professional physicians for diagnosed depression patients and normal subjects respectively.

[0079] In an optional embodiment, for the light fine-tuning of the multi-modal large model, the fine-tuning target is not only the identification of the emotional state of the target object, but also the identification of the learning emotional state of the target object, for example, in the online teaching scenario, the students are taken as the target objects, and the learning states of the students are identified by analyzing the multi-modal features such as facial expressions, voice tones, and writing behaviors of the students in the learning process as the analysis basis of the multi-modal large model. In this scenario, the positive samples can be individual data of students with abnormal learning states (e.g., inattention, low interactive participation, etc.), and the negative samples can be individual data of students with normal learning states (e.g., high attention or high interactive participation, etc.), and the positive and negative samples carry the description of the reasoning texts or evaluation logic of the teachers for judging different learning states.

[0080] In an embodiment, the terminal performs LoRA fine-tuning training on the pre-trained large language model through the positive and negative samples, and constructs a preset mapping dictionary of "visual features to symptom terms" according to the judgment basis written by professional physicians for diagnosed depression patients and normal subjects respectively.

[0081] After reading the target data to be analyzed, the terminal inputs the target data to be analyzed into the multi-modal reasoning large model. The multi-modal reasoning large model first extracts features from the key frame images and target audio segments in the target data to be analyzed, obtains feature embeddings of different modalities, and maps the feature embeddings of different modalities to a unified semantic space to obtain fused features, thereby realizing feature fusion of multi-modal data in the target data to be analyzed. Further, the multi-modal reasoning large model constructs the correlation between modalities in the fused features in the internal self-attention mechanism according to the semantic understanding and reasoning ability after pre-training and fine-tuning, and performs semantic understanding and reasoning on the fused features in combination with the correlation, and gradually generates a dynamic reasoning path corresponding to the current target object. The dynamic reasoning path can be in the form of a thinking chain. For example, the dynamic reasoning path can be "frequent appearance of 'not interesting' and 'tired' and the like in the target audio indicates that the patient may have low mood, and voice analysis shows slow speech speed and flat tone, which further supports the hypothesis of emotional inhibition; facial expression analysis in the key frame image shows drooping mouth corners and eye avoidance, which is consistent with the external characteristics of negative emotions".

[0082] In step 106, the multi-modal reasoning large model is used to analyze the target data to be analyzed in combination with the guidance of the dynamic reasoning path, to obtain the recognition result of the target object, and generate the target explanatory description text according to the dynamic reasoning path and the preset mapping dictionary.

[0083] In the embodiment of the present application, the terminal analyzes and reasons the fused features of the target data to be analyzed in combination with the guidance of the dynamic reasoning path by using the reasoning mapping ability learned by the multi-modal reasoning large model in the fine-tuning, to determine the recognition result of the current target object. The recognition result is a binary classification label of whether there is negative emotion, for example, the recognition result is "normal" or "depression". Then, since the dynamic reasoning path is the basis for classification of the multi-modal reasoning large model (which can be direct or indirect basis), the multi-modal reasoning large model can also generate an initial explanatory description text according to the dynamic reasoning path. However, the description method and the words used in the initial explanatory description text may have problems such as colloquialization. On this basis, the multi-modal reasoning large model further adjusts the description method in the initial explanatory description text according to the preset mapping dictionary, to obtain the target explanatory description text in the form of symptom terminology description.

[0084] In a specific embodiment, the target explanatory description text can be "Visual analysis: multiple close-ups of the woman's face appear in the picture, with obvious tear marks under her eyes, a drooping mouth, and relaxed orbicularis oculi muscles, showing a sad emotion. Voice analysis: the woman's voice is crying multiple times, with a slow speech rate and a low-pitched voice, expressing a sad and upset emotion; environmental clues: the environment is dimly lit, and the background items are arranged irregularly, which may indicate that the woman is in a more oppressive environment. Core symptoms: persistent sad micro-expressions (drooping mouth + relaxed orbicularis oculi muscles, etc.). Additional symptoms: sobbing, choking, low-pitched voice, sad, upset, negative words such as degrading; reduced hand gestures (<5 times per minute); delayed response (average reaction time > 3 seconds); self-deprecating vocabulary frequency; signs of circadian rhythm disorder (e.g., abnormal pupil diameter at night). Therefore, by supplementing the recognition result output by the multi-modal reasoning large model with the target explanatory description text, the explainability of the multi-modal large model output content is improved, which can enhance the degree of trust of the user (e.g., a psychologist or a patient or a family member) in the recognition result, making the multi-modal large model have more practical application value for identifying the state type of the target object, for example, in the intelligent auxiliary assessment of mental health (e.g., depression recognition or anxiety recognition, etc. psychological state recognition) or learning emotion state monitoring of the target object, which has more practical application prospects and significant popularization value and social significance.

[0085] In the above object recognition method, the dynamic reasoning path reflecting the characteristics of the target object in the current target data to be analyzed can be captured by the multi-modal reasoning large model for reasoning and analysis of the target data to be analyzed, which can not only serve as a guide for the multi-modal reasoning large model to identify and reason the target object, but also serve as a basis (which can be direct or indirect) for generating the target explanatory description text, so that the reasoning of the recognition result is strongly associated with the target explanatory description text, and therefore the target explanatory description text can reflect the reasoning logic of the recognition result. Further, the target explanatory description text is used as a supplement to the output result of the multi-modal reasoning large model, which enriches the dimensions of the object analysis result through the target explanatory description text, and further improves the explainability of the object analysis result.

[0086] In an exemplary embodiment, the target data to be analyzed includes key frame images and target audio segments, and the key frame images and the target audio segments need to be aligned before the multi-modal reasoning large model performs reasoning analysis, as shown in Figure 2 Step 102 includes steps 202 to 206. Among them:

[0087] Step 202, obtaining original data to be analyzed.

[0088] The original data to be analyzed includes video data and audio data.

[0089] In the embodiments of the present application, the terminal obtains original to-be-analyzed data in the corresponding storage space. The original to-be-analyzed data specifically refers to original audio and video records collected by a camera device (for example, a smart phone, a tablet computer, a smart camera, etc.) without compression distortion or cropping processing in the standardized object negative emotional disorder or negative emotional state recognition scenario. For example, the object recognition task can be to evaluate the psychological state of the target object in the psychological interview scene. At this time, the original to-be-analyzed data can be audio and video data collected from the target object in the clinical inquiry in the psychological interview scene or the psychological health screening. Alternatively, the object recognition task can also be to evaluate the learning state of the target object in the online learning monitoring scene, and the original to-be-analyzed data can be audio and video data recording the learning process of the target object.

[0090] The video data refers to a continuous visual information stream recorded in the interview or online learning process, stored in the form of a video file, and contains non-verbal behavior clues such as facial expressions, eye movements, body movements, and posture changes of the patient. For example, the video data can be a patient interview video with a video format of 1080p, 30fps.

[0091] The audio data can refer to a synchronous sound information stream recorded in the interview process, containing acoustic parameters such as the speech content, tone, speed, pause, and volume of the target object. For example, the audio data can be an interview audio with an audio format of 16kHz, containing voice fragments such as the patient's answers to questions, spontaneous statements, sighs, and silences.

[0092] In step 204, key frames are extracted from the video data according to a preset sampling strategy and a preset time window, and key frame images corresponding to each preset time window are obtained.

[0093] In the embodiments of the present application, since the video data is usually recorded at a high frame rate (for example, 25-30fps), the adjacent frames are highly similar in visual content, especially in relatively static scenes such as patient interviews, the facial expressions and posture changes are slow, and a large number of intermediate frames do not introduce new behavior or emotional clues. If all video frames are input into the multi-modal inference large model, not only the computing power will be wasted, but also the model may be disturbed by redundant information in extracting key behavior features.

[0094] Therefore, the terminal can extract key frames from the video data in each time window according to the pre-set preset sampling strategy and preset time window, obtain key frame images, and use the key frame images as representatives of the visual features in the preset time window. The multi-modal inference large model can analyze and infer according to the sampled key frame images in each preset time window, thereby reducing the amount of data to be processed by the multi-modal inference large model.

[0095] The preset sampling strategy can be fixed sampling frequency for key frame extraction. For example, the preset time window can be 1 second, and the fixed default sampling frequency can be 1 frame (e.g., the 15th frame) of image collected in each 1-second preset time window as a key frame image, thereby obtaining a key frame image for each preset time window. Alternatively, the preset sampling strategy can also be dynamic adjustment of the sampling frequency. For example, the preset sampling strategy can be dynamic adjustment of the default sampling frequency according to the feature type of the audio data, the complexity or change degree of the video frame, or the preliminary recognition result.

[0096] In step 206, the audio data is divided based on the preset time window to obtain a target audio segment corresponding to each preset time window.

[0097] In the embodiments of the present application, the terminal divides the audio data based on the preset time window to segment the audio data, thereby obtaining a plurality of target audio segments, so that the target audio segments are aligned with the key frame images on the timeline, and the semantic association between the multi-modalities is maintained. For example, the terminal divides the audio data into a plurality of target audio segments of "0th second to 1st second, 1st second to 2nd second,... " according to the preset time window of every 1 second, and the preset time windows of "0th second to 1st second" and "1st second to 2nd second" have one or more corresponding key frame images, respectively, so that each preset time window has a corresponding target audio segment and key frame image, thereby realizing multi-modal alignment of the target audio segment and the key frame image.

[0098] In the embodiments, the video data and the audio data in the original data to be analyzed are respectively sparsely sampled and segmented by the preset sampling strategy and the preset time window, thereby avoiding repeated calculation of redundant frames in the video. Moreover, the audio data is segmented to be aligned with the time points of the key frame images, thereby ensuring synchronization of the time sequence information of the key frame images and the audio data, enhancing the effectiveness of multi-modality fusion, and thereby improving the accuracy and robustness of the overall object recognition, thereby reducing the computational burden and resource consumption of the multi-modality inference large model under the premise of ensuring the integrity of the time sequence semantics.

[0099] In one exemplary embodiment, for the extraction of key frame images, dynamic sampling frequency division can be performed according to different time period types, and extraction can be performed according to the dynamic sampling frequency, as shown in Figure 3 Step 204 includes steps 302 to 304. Wherein:

[0100] In step 302, the feature type of the audio data is identified to determine the feature type of each time period in the audio data.

[0101] In the embodiments of the present application, the emotional state of the patient (e.g., anxiety, depression, cognitive load, or emotional fluctuation) can not only be reflected in the speech features (e.g., volume fluctuation, speech speed pause, pitch change, etc.) of the audio data, but also be synchronously reflected in non-verbal visual behaviors (e.g., subtle changes in facial expressions, blinking frequency, head posture, body micro-movement, etc.). Therefore, the audio data and the video data reflect the emotional state of the patient synchronously, and then the terminal can take the feature type of the audio data as the basis for adjusting the sampling frequency of the video data. For example, when the patient is emotionally excited, the speech energy is significantly enhanced, and is often accompanied by frowning, eye blinking, or body leaning forward; and when the patient is in a state of slow thinking or emotional depression, the speech tends to be monotonous, the silence increases, and the facial muscle activity also significantly decreases. Therefore, the acoustic features carried in the audio data can also synchronously reflect the activity level of the visual behavior.

[0102] Therefore, the terminal performs fine-grained analysis on the original audio data according to a preset time window (e.g., 1 second), identifies the acoustic feature type of the speech in each time period, and thus provides a basis for subsequent dynamic adjustment of the sampling frequency of the key frame image. The acoustic feature type includes speech energy fluctuation, silence ratio, and fundamental frequency change rate. For example, in the time period of 1st to 10th second, the speech energy fluctuation is of a low energy fluctuation type, and thus the feature type of the time period is determined as a low energy fluctuation.

[0103] The speech energy fluctuation is used to measure the change degree of the amplitude of the speech signal in a unit of time, and reflects the emotional excitement level of the patient when speaking. The silence ratio refers to the proportion of the audio data whose speech activity detection result is no speech in a preset time window of 1 second. The fundamental frequency change rate is used to reflect the pitch of the speech in the audio data, and the fundamental frequency change rate can measure the rising and falling speed of the pitch in a unit of time.

[0104] The feature type corresponding to the speech energy fluctuation includes high energy fluctuation and low energy fluctuation. When the patient is emotionally excited, anxious, or has a strong emotional response, the speech energy is relatively intense, and vice versa, if the emotional state of the patient is stable, the speech energy fluctuation is relatively gentle. Therefore, the terminal can determine that the speech energy fluctuation is greater than or equal to 0.5 as high energy fluctuation, and determine that the speech energy is less than 0.5 as low energy fluctuation. The speech energy fluctuation can be obtained by calculating the intensity of the sound signal in each frame of the audio data to obtain an energy sequence that changes over time, and then performing normalization processing on the energy sequence. The normalized value obtained after normalization can be used as a score of the speech energy fluctuation, which is used to measure the intensity of the energy change of the speech signal in the time dimension, and reflects the stability of the emotional state of the target object.

[0105] The feature type corresponding to the silence ratio includes high activity and low activity. If the proportion of silence in a time period is large, the feature type of the time period is determined as low activity; if the proportion of silence in a time period is small, the feature type of the time period is determined as high activity.

[0106] The feature type of the fundamental frequency change rate includes high fundamental frequency change rate and low fundamental frequency change rate. The terminal can determine the feature type of the fundamental frequency change rate greater than 20 Hz / s (20 hertz per second) as high fundamental frequency change rate, and determine the feature type of the fundamental frequency change rate less than 20 Hz / s as low fundamental frequency change rate.

[0107] In step 304, the target sampling frequency of the video data in each preset time window is determined based on the feature type, and the video data is sampled according to the target sampling frequency to obtain the key frame image corresponding to each preset time window.

[0108] In the embodiment of the application, the terminal takes the feature type of each time period in the audio data as the feature type of the video data, and dynamically allocates the target sampling frequency of the video data in each preset time window according to the feature type, so as to realize dynamic sparse sampling of the target video, and realize increasing the sampling density in each preset time window in the information-rich period to capture the subtle behavior changes of the patient, and reducing the sampling frequency in each preset time window in the information-sparse period to avoid information redundancy and reduce the computing cost. Further, the terminal samples the video data in each preset time window according to the target sampling frequency to obtain the key frame image required for identifying and analyzing the video data.

[0109] Specifically, the sharp fluctuation of speech energy is usually accompanied by rapid non-verbal behaviors such as facial muscle tension, eye movement change, head micro-motion, etc. For example, when the patient expresses pain or excitement, he / she may frown, blink faster or lean forward. In the default sampling frequency, if 1 frame is sampled in the sampling time window of 1s, these instantaneous expressions may be missed, therefore, the sampling density needs to be increased to cover the entire fluctuation process, so as to ensure the synchronous modeling of visual features and speech emotion. Therefore, if the speech energy fluctuation of the current time period is greater than 0.5, it means that the feature type of the current time period is high speech energy fluctuation, and the terminal increases the sampling frequency in the time period. If the default sampling frequency is 1 frame per second, the terminal increases the target sampling frequency in the time period to 3 frames per second, for example, taking 30 frames of video data per second as an example, the terminal takes the 10th frame, the 20th frame and the 30th frame as key frame images.

[0110] For the feature type of low activity with a low proportion of silence, i.e., the proportion of silence without speech is greater than 70%, it indicates that the patient keeps silent for a long time, and may be in a state of gazing, immobility, etc., which indicates that the difference between the images in the video data at this time is small. Therefore, the terminal adjusts the sampling frequency of the period of the low activity feature type to keep 1 frame per second, for example, the 15th frame in each 1-second preset time period is taken as a key frame image, avoiding the introduction of a large number of repeated video frames, and thereby saving the calculation resource consumption of the multi-modal reasoning large model for analysis and reasoning.

[0111] If the terminal identifies that the pitch variation rate of a certain period of audio data is a high pitch variation rate, i.e., the pitch variation rate is greater than 20 Hz / s, it indicates that there is a situation of semantic emphasis, emotional turning point or increased cognitive load in the audio data in this period, at this time, the patient may have facial movements such as mouth corner micro-movement and eyebrow raising in the video data in this period, at this time, the terminal increases the sampling frequency to 2 frames per second on the basis of the default sampling frequency of 1 frame per second, for example, the 10th frame and the 25th frame in each 1-second preset time period are taken as key frame images, in order to capture the visual features corresponding to the pitch change start and peak value.

[0112] In a specific embodiment, the terminal independently executes the target sampling strategy for each 1-second preset time window in combination with the time stamp of the feature type of the audio data when parsing the video stream. For example, if the 3-4 second window is determined to be a high energy fluctuation, the terminal extracts the 85th, 95th and 105th frames (corresponding to the global 10th, 20th and 30th frame positions) from the video frames (for example, the 76-105 frames, assuming 30fps) corresponding to the window as key frame images. In addition, if a certain preset time window is determined to be both a high energy fluctuation and a high pitch variation rate, the terminal takes the higher sampling frequency as the target sampling frequency, avoiding the omission of features in the video data.

[0113] In this embodiment, by enhancing visual sampling in emotional or cognitive key periods, the subtle behavior signs reflecting negative emotions are avoided to be omitted, and by reducing redundant frame processing in low information periods, the repeated calculation of redundant frames and invalid speech is avoided, the GPU (Graphics Processing Unit, graphics processor) memory occupation and calculation burden are greatly reduced, and the multi-modal reasoning large model is more suitable for deployment in edge devices or resource-limited medical scenarios.

[0114] In an exemplary embodiment, for the extraction of key frame images, in addition to the dynamic sampling frequency division by the type of different periods, the terminal can also dynamically adjust the global default sampling frequency based on the initial identification result of the original analysis data, and sample and extract key frame images according to the adjusted target sampling frequency, such as Figure 4As shown, step 204 includes steps 402 to 406. Among them:

[0115] Step 402, sampling the video data in the original analysis data according to the default sampling frequency to obtain initial key frame images of each preset time window, and determining the initial audio segment corresponding to the initial key frame image.

[0116] In the embodiment of the application, after receiving the original audio and video interview data of the target object (patient to be identified for emotion), the terminal preliminarily samples the video data using a default sampling frequency (for example, 1 frame per second). The sampling process is carried out in units of a preset time window (for example, 1 second per window), ensuring the time sequence continuity covering the entire interview process. For each preset time window, the terminal extracts a corresponding number of initial key frame images (for example, 1 frame in a preset time window of 1 second), and synchronously intercepts an initial audio segment aligned with the timestamp of the initial key frame image, forming a "image-audio" paired data unit as the basis for subsequent preliminary analysis.

[0117] Step 404, obtaining an initial recognition result based on the initial key frame image and the initial audio segment.

[0118] In the embodiment of the application, the terminal inputs the initial key frame image and the corresponding initial audio segment into the multi-modal inference large model, and preliminarily evaluates the initial key frame image and the initial audio segment for negative emotional disorder or negative emotional state through the initial recognition result, to obtain the initial recognition result. Taking depression recognition as an example, the multi-modal inference large model fuses the visual features (for example, facial expression intensity, eye direction) of the initial key frame image and the speech features (for example, fundamental frequency, speech rate, keyword frequency) of the initial audio segment, determines a preliminary depression tendency score as the initial recognition result, for example, the value range of the initial recognition result is 0 to 10 points. Among them, 0 points represent no depression signs, and 10 points represent high suspicion of severe depression.

[0119] Step 406, dynamically adjusting the default sampling frequency according to the initial recognition result to obtain a target sampling frequency, and sampling the video data according to the target sampling frequency to obtain a key frame image corresponding to each preset time window.

[0120] In the embodiments of the present application, the terminal determines the adjustment strategy for dynamic adjustment of the default sampling frequency based on the initial recognition result according to the hierarchical adjustment strategy. Specifically, the hierarchical adjustment strategy includes a low-risk adjustment strategy, a medium-risk adjustment strategy, and a high-risk adjustment strategy, and the hierarchical adjustment strategy includes a plurality of preset score thresholds for dividing the risk classification of the current video data. In some embodiments, the terminal determines the target adjustment strategy corresponding to the initial recognition result according to the preset score threshold corresponding to each hierarchical risk adjustment strategy, wherein the low-risk adjustment strategy is that when the initial recognition result satisfies a first condition corresponding to a low-risk score threshold, the sampling adjustment strategy is to keep the default sampling strategy as the target sampling strategy; the medium-risk adjustment strategy is that when the initial recognition result satisfies a second condition corresponding to a medium-risk score threshold, the sampling adjustment strategy is to adjust the sampling frequency according to a preset frequency increasing rule; and the high-risk adjustment strategy is that when the initial recognition result satisfies a third condition corresponding to a high-risk score threshold, the sampling adjustment strategy is to further increase the sampling frequency and take a high sampling frequency as the target sampling frequency. The first condition, the second condition, and the third condition can be flexibly set according to business requirements. For example, if the initial recognition result is a depression tendency score less than 3 points, the depression tendency of the patient in the video data is low risk, and the low-risk adjustment strategy is to keep the default sampling frequency as the target sampling frequency; if the initial recognition result is a depression tendency score between 3 points and 7 points, the depression tendency of the patient in the video data is medium risk, and the medium-risk adjustment strategy is to increase the sampling frequency to 2 frames per second as the target sampling frequency, so as to enhance the capture ability of the voice pause mode (for example, answer delay, silence duration) and micro-expression change (for example, short mouth corner pull-down, frown); and if the initial recognition result is a depression tendency score greater than 7 points, the depression tendency of the patient in the video data is high risk, and the high-risk adjustment strategy is to increase the sampling frequency to 3 frames per second and preferentially extract close-up frames containing eye regions for analyzing visual cues such as heavy depression highly related to heavy depression, such as drooping eyelids, pupil lag, and tear light.

[0121] In some embodiments, the terminal re-samples the video data according to the adjusted target sampling frequency, and in the case of medium risk and high risk, more dense key frame images that can better reflect the negative emotional state are obtained under each preset time window. The key images obtained after re-sampling and their corresponding audio segments will be the target data to be analyzed and input into the complete multi-modal inference large model to generate a dynamic inference path and a final state recognition result.

[0122] In this embodiment, the adjustment strategy for dynamically adjusting the default sampling frequency is determined through the initial recognition result, and the default sampling frequency is dynamically adjusted according to the adjustment strategy, thereby realizing dynamic adjustment of the sampling frequency and adaptive adjustment from coarse-grained sampling to fine-grained sampling. On the basis of ensuring the accuracy of object recognition, the computing resources are dynamically adjusted, so that the optimal balance between recognition efficiency and recognition accuracy is achieved, the resource utilization rate and the recognition gain value (i.e., the ratio of recognition accuracy to computing resources) are improved, and the efficiency of object recognition can be improved.

[0123] In one exemplary embodiment, the multi-task output head of the multi-modal reasoning large model includes a classification head and a text generation head; as Figure 5 As shown, step 106 includes steps 502 to 506. Among them:

[0124] Step 502, under the guidance of the dynamic reasoning path, the target to be analyzed data is analyzed based on the classification head, and the recognition result of the target object is obtained.

[0125] In the embodiment of the application, the terminal inputs the generated target to be analyzed data into the multi-modal reasoning large model, and the target to be analyzed data is a high information density key frame image and its corresponding target audio segment extracted according to the dynamic sampling strategy, wherein the backbone network of the multi-modal reasoning large model can be an open source pre-training reasoning large model. First, the multi-modal reasoning large model extracts deep features and cross-modal alignment of visual and auditory modalities. Then, the multi-modal reasoning large model outputs a structured recognition result through the classification head, and the recognition result determines that the target object is in a normal state or has a negative emotional tendency.

[0126] Step 504, generating an initial explanatory description text corresponding to the recognition result according to the text generation head and the dynamic reasoning path.

[0127] In the embodiments of the present application, after receiving the cross-modal context representation output by the backbone network, the text generation head of the multi-modal reasoning large model inputs the dynamic reasoning path as guide information into the text generation head. The text generation head is guided by the dynamic reasoning path to control the attention distribution and weight of the context output by the backbone network, dynamically adjust the output probability distribution of the language model, and automatically generate an initial explanatory description text in natural language form. The dynamic reasoning path is obtained by the multi-modal reasoning large model analyzing the target data to be analyzed. The dynamic reasoning path can be composed of a structured and interpretable intermediate decision state sequence. The intermediate decision state sequence is a state record generated by the multi-modal reasoning large model during the reasoning process. Each state includes the modality of the target data to be analyzed, the activated subnetwork during the model reasoning process, the key feature identifier corresponding to the target data to be analyzed, and the intermediate judgment result. For example, during the reasoning analysis of the multi-modal reasoning large model on the target data to be analyzed, for the part of the data to be analyzed with a time of 0 to 30 seconds, the multi-modal reasoning large model analyzes the input voice modality and video modality, activates the speech speed detection subnetwork, identifies the speech speed as the key feature identifier, and judges that the state result of this part of data is speech retardation. Similarly, the multi-modal reasoning large model can activate different feature subnetworks for other parts of the data to be analyzed and for different modal data. For example, the feature subnetwork can also be an expression recognition subnetwork, an emotion recognition subnetwork, an emotion risk recognition subnetwork, etc. Each feature subnetwork has a corresponding key feature, and each feature subnetwork can obtain a local judgment result corresponding to the key feature according to the key feature.

[0128] In some embodiments, the implementation of the dynamic reasoning path's control over the attention distribution and weight of the context output by the backbone network can be that, in the process of generating natural language explanation text by the text generation head, the attention and weight control logic learned by the multi-modal reasoning large model in the model training process is combined with the state records in the dynamic reasoning path to determine the adjustment strategy corresponding to each state record, and the attention distribution and weight of the context output by the backbone network are dynamically adjusted through the adjustment strategy. For example, when the dynamic reasoning path is "t=30-60s video modality, the key feature detected by the expression recognition subnetwork is frown intensity of 0.8", the text generation head improves the attention weight of the context vector of the target data to be analyzed in the time period (t=30-60s) and the corresponding modality channel (video modality) in the backbone network when calculating the attention distribution, so that it is the main feature in the context vector obtained by weighted summation; at the same time, for the context segments that are not marked as key features by the dynamic reasoning path (for example, background noise area or irrelevant action), their corresponding attention weights are actively suppressed, so that the initial explainable description of natural language is generated according to the main feature.

[0129] The dynamic reasoning path is not only used to guide the multi-modal reasoning large model to perform structured identification on the target data to be analyzed, but also used to generate an initial explainable description text in combination with the text generation head, so as to ensure that the identification result and the explainable description are highly consistent in semantic logic, focus and reasoning level.

[0130] In one embodiment, the dynamic reasoning path is used to regulate the feature subnetwork activated by the multi-modal reasoning large model in the classification task, focus on which modality clues (such as speech pauses, facial action unit changes, etc.), and then determine the final negative emotion tendency recognition result. At this time, when the dynamic reasoning path is applied to the text generation head synchronously, the dynamic reasoning path serves as a context guide signal to control the attention distribution and word generation preference in the decoding process of the multi-modal reasoning large model, so that the generated initial explainable description text strictly surrounds the features relied on in the classification stage, ensuring that the generated initial explainable description is a true reflection of the recognition result.

[0131] Further, the terminal embeds the recognition result of the target object as a soft prompt into the dynamic reasoning path to obtain a prompt word for generating an explanatory description text, and generates an initial explanatory description text corresponding to the current recognition result according to the reasoning capability of the multi-modal reasoning large model. For example, if the recognition result of the target object is "severe depressive tendency", the multi-modal reasoning large model encodes the recognition result of "severe depressive tendency" as a continuous vector, embeds it as a soft prompt into the dynamic reasoning path, and inputs the dynamic reasoning path into the text generation head. The text generation head generates reasoning text that recognizes the target object as "severe depressive tendency" as an initial explanatory description text under the guidance of the dynamic reasoning path by learning the explanatory text generation capability through fine-tuning training.

[0132] In step 506, the initial explanatory description text is mapped according to a preset mapping dictionary to obtain a target explanatory description text.

[0133] In the embodiment of the application, in order to improve the clinical applicability and understandability of the output text, the terminal performs term standardization processing on the generated initial explanatory description text through a preset mapping dictionary constructed in advance. The preset mapping dictionary defines mapping rules from technical expressions to clinical standard languages, for example, mapping "blink frequency lower than the norm by 30%" to "psychomotor inhibition, manifested as dull eyes and slow reaction". The terminal first locates non-standard terms in the initial explanatory description text, and selects the most matched clinical expression in combination with the context semantics to ensure that the mapping result conforms to the context. Further, after completing the term mapping, the target explanatory description text output by the terminal ensures the term standardization of the target explanatory description text.

[0134] In the embodiment, through the dynamic reasoning path, not only the existence of negative emotions or normal state of the target object is accurately recognized based on the key frame image and the target audio segment, but also the initial explanatory description text strictly aligned with the recognition result in terms of semantic logic and reasoning basis is generated synchronously, avoiding the problem that the target explanatory description text is out of line with the recognition result, improving the explainability of the recognition result, and further converting the initial explanatory description text into a term expression conforming to the clinical standard through the preset mapping dictionary, thereby improving the accuracy of the target explanatory description text.

[0135] In one exemplary embodiment, as shown in Figure 6 If the recognition result is an abnormal result, step 106 includes steps 602 to 604. Among them:

[0136] In step 602, the multi-modal reasoning large model performs reasoning analysis on the description text corresponding to each preset dimension in the target explanatory description text to obtain an initial analysis result corresponding to each preset dimension.

[0137] In the embodiments of the present application, the multi-modal reasoning large model further includes a regression head, and the terminal inputs the target explanatory description text into the regression head of the multi-modal reasoning large model. The regression head respectively performs fine-grained semantic analysis and quantitative mapping for the preset clinical dimensions to obtain an initial analysis result corresponding to each preset dimension, where the initial analysis result can be an initial score. Specifically, the preset dimensions include visual analysis, speech analysis, environmental clues, core symptoms, and additional symptoms. The multi-modal reasoning large model first extracts symptom description fragments related to each dimension from the target descriptive text, for example, "eyebrows are continuously knitted, and eyes are avoided" is classified into the visual analysis dimension, "speech speed is significantly slowed down, and frequent pauses" are classified into the speech analysis dimension, "being alone in a room, dim light, and messy objects" are classified into the environmental clues dimension, "persistent low mood and loss of interest" are classified into the core symptom dimension, and "sleep disorder and decreased appetite" are classified into the additional symptom dimension. Subsequently, the multi-modal reasoning large model performs an initial score of 0 to 10 for the description content of each preset dimension based on the mapping relationship between the clinical semantics and the scores learned in the pre-training stage, and the scoring process comprehensively considers the frequency, intensity, persistence, and clinical significance of the symptoms to obtain the initial score corresponding to each preset dimension, which reflects the independent severity of each dimension in the current target object and provides a structured basis for subsequent comprehensive evaluation.

[0138] In step 604, each initial analysis result is processed according to a preset weight corresponding to each preset dimension to obtain an analysis result tendency degree of the target object.

[0139] In the embodiments of the present application, the terminal performs weighted summation processing on the initial scores of each dimension according to the preset weights corresponding to each preset dimension to calculate the analysis result tendency degree of the target object, which is the symptom severity score of the negative emotional tendency of the target object, for example, representing the overall depression degree of the target object. Specifically, taking the preset weights of visual analysis 30%, speech analysis 25%, environmental clues 15%, core symptoms 20%, and additional symptoms 10% as an example, the total score = visual score x 0.30 + speech score x 0.25 + environmental score x 0.15 + core symptom score x 0.20 + additional symptom score x 0.10, and the result is rounded to one decimal place, strictly limited in the range of 0 to 10. The weighting mechanism reflects the differentiated attention degree to different information sources in clinical evaluation, for example, visual and speech as direct behavior indicators have higher weights, environmental clues as auxiliary context evidence have moderate weights, and core symptoms are given key weights due to their evaluation specificity.

[0140] In this embodiment, the initial score is generated by the multi-modal reasoning large model under multiple preset dimensions, and the symptom degree result is obtained by weighted summation calculation according to the preset weight, so as to realize the quantification of the negative emotion tendency degree of the target object, and ensure that the final symptom degree result can reflect the patient's objective behavior performance contained in the video data and audio data in the target data to be analyzed, so as to provide reliable, interpretable and comparable quantitative basis for intelligent recognition of negative emotion, and improve the authenticity and accuracy of the recognition result and the target explanation description text.

[0141] In one example embodiment, as shown in Figure 7 there is provided an example of an object recognition method, wherein:

[0142] Step 701, acquiring original input data (video with sound).

[0143] Step 702, dynamically sparse sampling the input data.

[0144] Step 7021, key frame extraction of the input data according to a sampling frequency of 1 frame per second.

[0145] Step 7022, segmenting the audio and aligning it with the key frame time.

[0146] Step 703, loading the LoRA fine-tuned multi-modal large model.

[0147] Step 7031, obtaining the recognition result of depression or normal according to the multi-modal large model classification output.

[0148] Step 7032, obtaining the interpretable reasoning result according to the multi-modal large model classification output.

[0149] Step 704, displaying the interpretable reasoning result and the recognition result as auxiliary recognition results.

[0150] It should be understood that although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0151] Based on the same inventive concept, the embodiments of the present application also provide an object recognition device for implementing the above-mentioned object recognition method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more object recognition device embodiments provided below can refer to the limitations of the object recognition method in the above text, which will not be repeated here.

[0152] In one exemplary embodiment, as shown in Figure 8 An object recognition device 800 is provided, comprising: an acquisition module 801, a first inference module 802, and a second inference module 803, wherein:

[0153] The acquisition module 801 is configured to acquire target analysis data containing a target object;

[0154] The first inference module 802 is configured to perform inference analysis on the target analysis data according to a multi-modal inference large model, to obtain a dynamic inference path of the target object;

[0155] The second inference module 803 is configured to perform inference on the target analysis data through the multi-modal inference large model under the guidance of the dynamic inference path, to obtain an identification result of the target object, and to generate a target explanatory description text according to the dynamic inference path and a preset mapping dictionary.

[0156] In one embodiment, the target analysis data includes key frame images and a target audio segment; the acquisition module 801 is specifically configured to acquire original analysis data; the original analysis data includes video data and audio data;

[0157] The video data is subjected to key frame extraction according to a preset sampling strategy and a preset time window, to obtain a key frame image corresponding to each preset time window;

[0158] The audio data is divided based on a preset time window to obtain a target audio segment corresponding to each preset time window.

[0159] In one of the embodiments, the obtaining module 801 is specifically configured to perform feature type identification on the audio data, and determine feature types of time periods in the audio data.

[0160] Based on the feature types, a target sampling frequency of the video data in each preset time window is determined, and the video data is sampled based on the target sampling frequency to obtain a key frame image corresponding to each preset time window.

[0161] In one of the embodiments, the obtaining module 801 is specifically configured to sample the video data in the original analysis data based on a default sampling frequency to obtain initial key frame images of the preset time windows, and determine initial audio segments corresponding to the initial key frame images.

[0162] Based on the initial key frame images and the initial audio segments, an initial recognition result is obtained.

[0163] The default sampling frequency is dynamically adjusted based on the initial recognition result to obtain a target sampling frequency, and the video data is sampled based on the target sampling frequency to obtain a key frame image corresponding to each preset time window.

[0164] In one of the embodiments, the multi-task output head of the multi-modal large inference model includes a classification head and a text generation head; the second inference module 803 is specifically configured to perform inference analysis on the target analysis data based on the classification head under the guidance of the dynamic inference path to obtain a recognition result of the target object.

[0165] An initial explanatory description text corresponding to the recognition result is generated based on the text generation head and the dynamic inference path.

[0166] The initial explanatory description text is mapped based on a preset mapping dictionary to obtain a target explanatory description text.

[0167] In one of the embodiments, if the recognition result is an abnormal result, the apparatus 800 further includes:

[0168] The third inference module is configured to perform inference analysis on description texts corresponding to each preset dimension in the target explanatory description text based on the multi-modal large inference model to obtain an initial analysis result corresponding to each preset dimension.

[0169] The calculation module is configured to process the initial analysis results based on preset weights corresponding to the preset dimensions to obtain an analysis result tendency degree of the target object.

[0170] Each module in the aforementioned object recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0171] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an object recognition method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0172] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0173] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0174] Obtain the target data to be analyzed, which contains the target object;

[0175] According to the multi-modal reasoning large model, the target to-be-analyzed data is analyzed to obtain a dynamic reasoning path of the target object;

[0176] According to the guidance of the dynamic reasoning path, the target to-be-analyzed data is analyzed by the multi-modal reasoning large model to obtain an identification result of the target object, and a target explanatory description text is generated according to the dynamic reasoning path and a preset mapping dictionary.

[0177] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0178] Obtaining original to-be-analyzed data; the original to-be-analyzed data includes video data and audio data;

[0179] According to a preset sampling strategy and a preset time window, key frame images corresponding to each preset time window are obtained by extracting key frames from the video data.

[0180] Based on the preset time window, a target audio segment corresponding to each preset time window is obtained by dividing the audio data.

[0181] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0182] Identifying the feature type of the audio data to determine the feature type of each time period in the audio data;

[0183] Based on the feature type, a target sampling frequency of the video data in each preset time window is determined, and the video data is sampled according to the target sampling frequency to obtain key frame images corresponding to each preset time window.

[0184] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0185] According to a default sampling frequency, initial key frame images of each preset time window are obtained by sampling the video data in the original to-be-analyzed data, and initial audio segments corresponding to the initial key frame images are determined;

[0186] Based on the initial key frame images and the initial audio segments, an initial identification result is obtained;

[0187] According to the initial identification result, the default sampling frequency is dynamically adjusted to obtain a target sampling frequency, and the video data is sampled according to the target sampling frequency to obtain key frame images corresponding to each preset time window.

[0188] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0189] Under the guidance of the dynamic inference path, the target to-be-analyzed data is analyzed based on the classification head to obtain an identification result of the target object;

[0190] According to the text generation head and the dynamic inference path, an initial explanatory description text corresponding to the identification result is generated;

[0191] According to a preset mapping dictionary, the initial explanatory description text is mapped to obtain a target explanatory description text.

[0192] In one embodiment, the processor also implements the following steps when executing the computer program:

[0193] According to the multi-modal inference large model, the description text corresponding to each preset dimension in the target explanatory description text is analyzed to obtain an initial analysis result corresponding to each preset dimension;

[0194] According to the preset weight corresponding to each preset dimension, each initial analysis result is processed to obtain an analysis result tendency degree of the target object.

[0195] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps in each method embodiment described above.

[0196] In one embodiment, a computer program product is provided, and the computer program product includes a computer program. The computer program is executed by a processor to implement the steps in each method embodiment described above.

[0197] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0198] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0199] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0200] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An object recognition method, characterized in that, The method includes: Obtain the target data to be analyzed, which contains the target object; The target data to be analyzed is analyzed using a multimodal reasoning model to obtain the dynamic reasoning path of the target object. Guided by the dynamic reasoning path, the target data to be analyzed is reasoned through the multimodal reasoning model to obtain the identification result of the target object, and an explanatory description text of the target is generated based on the dynamic reasoning path and the preset mapping dictionary.

2. The method according to claim 1, characterized in that, The target data to be analyzed includes keyframe images and target audio segments; the process of obtaining the target data to be analyzed containing the target object includes: Obtain the raw data to be analyzed; the raw data to be analyzed includes video data and audio data. The video data is extracted based on a preset sampling strategy and a preset time window to obtain a key frame image corresponding to each preset time window. The audio data is segmented based on the preset time window to obtain the target audio segment corresponding to each preset time window.

3. The method according to claim 2, characterized in that, The step of extracting keyframes from the video data according to a preset sampling strategy and a preset time window to obtain a keyframe image corresponding to each preset time window includes: The audio data is subjected to feature type identification to determine the feature type of each time period in the audio data; Based on the feature type, the target sampling frequency of the video data within each preset time window is determined, and the video data is sampled according to the target sampling frequency to obtain the keyframe image corresponding to each preset time window.

4. The method according to claim 2, characterized in that, The step of extracting keyframes from the video data according to a preset sampling strategy and a preset time window to obtain a keyframe image corresponding to each preset time window includes: The video data in the original analysis data is sampled according to the default sampling frequency to obtain the initial keyframe image of each preset time window, and the initial audio segment corresponding to the initial keyframe image is determined. Based on the initial keyframe image and the initial audio segment, an initial recognition result is obtained; The default sampling frequency is dynamically adjusted based on the initial recognition result to obtain the target sampling frequency. The video data is then sampled based on the target sampling frequency to obtain the keyframe image corresponding to each preset time window.

5. The method according to claim 1, characterized in that, The multi-task output head of the multimodal reasoning model includes a classification head and a text generation head; guided by the dynamic reasoning path, the multimodal reasoning model infers the target data to be analyzed, obtains the recognition result of the target object, and generates explanatory descriptive text of the target based on the dynamic reasoning path and a preset mapping dictionary, including: Guided by the dynamic reasoning path, the target data to be analyzed is reasoned and analyzed based on the classification head to obtain the identification result of the target object; The initial explanatory description text corresponding to the recognition result is generated based on the text generation head and the dynamic reasoning path; The initial explanatory description text is mapped according to a preset mapping dictionary to obtain the target explanatory description text.

6. The method according to claim 1, characterized in that, If the identification result is an abnormal result, after generating the target explanatory description text based on the dynamic reasoning path and the preset mapping dictionary, the method further includes: Based on the multimodal reasoning big model, reasoning analysis is performed on the description text corresponding to each preset dimension in the target explanatory description text to obtain the initial analysis result corresponding to each preset dimension; The initial analysis results are processed according to the preset weights corresponding to each preset dimension to obtain the degree of tendency of the analysis results of the target object.

7. An object recognition device, characterized in that, The device includes: The acquisition module is used to acquire target data to be analyzed, which contains the target object. The first reasoning module is used to perform reasoning analysis on the target data to be analyzed based on the multimodal reasoning big model, and obtain the dynamic reasoning path of the target object; The second reasoning module is used to combine the guidance of the dynamic reasoning path, reason about the target data to be analyzed through the multimodal reasoning big model, obtain the identification result of the target object, and generate target explanatory description text according to the dynamic reasoning path and the preset mapping dictionary.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images