Behavior analysis method and device, electronic equipment and storage medium

Abstract

The invention provides a behavior analysis method and device, electronic equipment and a storage medium, and relates to the technical field of behavior analysis. The method comprises the steps of obtaining multi-modal data collected for a to-be-detected target, and performing behavior analysis training on a pre-training model based on the multi-modal data to obtain a multi-modal large model; performing behavior analysis on the multi-modal data based on a multi-modal large model to obtain behavior text description for the to-be-detected target; and performing statistical analysis on the behavior text description based on a large language model to obtain a behavior analysis result for the to-be-detected target. The problem of low efficiency of behavior analysis in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the technical field of behavior analysis, and specifically, to a behavior analysis method, device, electronic device and storage medium. Background Art

[0002] In recent years, quantitative analysis methods of animal behavior based on machine learning have developed rapidly. By obtaining animal videos to conduct refined behavioral analysis of animals, a series of scientific research and industrialization results have been produced.

[0003] However, existing animal behavior analysis methods mainly extract interesting body point features from animal videos, and perform behavior analysis through methods including numerical calculation, behavior machine learning classification, clustering, etc. Among them, the existing methods perform behavior analysis through behavior machine learning classification or clustering, and the analysis accuracy is limited. At the same time, the existing methods focus on a single small-scale data and cannot extract universal behavioral features. The experiment is difficult to reproduce. It is processed through multiple stages and takes a long time, resulting in low analysis efficiency.

[0004] It can be seen from the above that the problem of low efficiency of behavior analysis in the prior art still needs to be solved. Summary of the invention

[0005] The present application provides a behavior analysis method, device, electronic device and storage medium, which can solve the problem of low accuracy of behavior analysis in related technologies. The technical solution is as follows:

[0006] According to one aspect of the present application, a behavior analysis method includes:

[0007] Obtain multimodal data collected for the target to be detected, and perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a multimodal large model;

[0008] Performing behavioral analysis on the multimodal data based on a multimodal large model to obtain a behavioral text description of the target to be detected;

[0009] The behavior text description is statistically analyzed based on a large language model to obtain a behavior analysis result for the target to be detected.

[0010] According to one aspect of the present application, a behavior analysis device includes:

[0011] A data acquisition module is used to obtain multimodal data collected for the target to be detected, and to perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a multimodal large model;

[0012] A behavior analysis module, used to perform behavior analysis on the multimodal data based on a multimodal large model to obtain a text description of the behavior of the target to be detected;

[0013] The statistical analysis module is used to perform statistical analysis on the behavior text description based on a large language model to obtain a behavior analysis result for the target to be detected.

[0014] In an exemplary embodiment, the data acquisition module includes:

[0015] A data set unit, used for constructing a behavior semantic data set based on the multimodal data;

[0016] A training unit is used to perform behavior analysis training on the pre-trained model based on the behavior semantic data set to obtain a multimodal large model.

[0017] In an exemplary embodiment, the data set unit includes:

[0018] A clustering subunit, configured to perform cluster analysis on the multimodal data to divide the multimodal data into at least two short-term data segments;

[0019] The behavior semantics subunit is used to set corresponding behavior semantics for the short-term data segment to generate a behavior semantics data set.

[0020] In an exemplary embodiment, the training unit comprises:

[0021] A behavior analysis subunit, used for inputting the behavior semantic data set into the pre-trained model to perform behavior analysis, and obtaining a behavior text description for each short-term data segment in the behavior semantic data set;

[0022] An evaluation subunit, configured to evaluate the accuracy of the behavior text description based on the behavior semantics in the semantic data set, and obtain an evaluation result indicating the accuracy of the behavior description of the behavior semantics of the short-term data segment by the pre-trained model;

[0023] A parameter adjustment subunit is used to adjust the parameters of the pre-trained model based on the evaluation result and repeat the training until the behavior description accuracy of the behavior analysis of the pre-trained model reaches a preset threshold, thereby obtaining a multimodal large model.

[0024] In an exemplary embodiment, the behavior analysis module includes:

[0025] A question information unit, configured to generate corresponding question information based on the multimodal data;

[0026] A behavior text description unit is used to generate a behavior text description in a preset format corresponding to the multimodal data based on the problem information.

[0027] In an exemplary embodiment, the behavior text description unit includes:

[0028] A first text description subunit, configured to generate a corresponding behavior text description for each short-term data segment in the multimodal data;

[0029] The second text description subunit is used to generate a text description corresponding to the multimodal data behavior for the behavior sequence of each of the short-term data segments.

[0030] In an exemplary embodiment, the behavior text description includes behavior semantic text and motion parameter text;

[0031] The first text description subunit includes:

[0032] A body point subunit, used to determine the three-dimensional body point data corresponding to the target to be detected in the short-time data segment;

[0033] A motion parameter subunit, configured to predict motion parameters based on the three-dimensional body point data and generate motion parameter text;

[0034] The semantic prediction subunit is used to perform behavioral semantic prediction on the short-term data segment to generate behavioral semantic text.

[0035] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the behavior analysis method as described above.

[0036] According to one aspect of the present application, a storage medium stores computer-readable instructions thereon, and the computer-readable instructions are executed by one or more processors to implement the behavior analysis method as described above.

[0037] The beneficial effects of the technical solution provided by this application are:

[0038] In the above technical scheme, the present application trains a multimodal large model through multimodal data, generates a text description of behavior through the multimodal large model, and then generates an analysis result through a large language model, wherein the pre-training model is fine-tuned through multimodal data, and large-scale multimodal data can be learned, and the multimodal large model is allowed to better utilize large-scale data, thereby extracting general behavioral features and analysis, improving the accuracy of behavior classification, recognition, and description, and obtaining a multimodal large model with high behavior description accuracy, which can effectively improve the efficiency of behavior analysis. Through the joint use of a multimodal large model and a large language model, behavior analysis of multimodal data is performed in an end-to-end manner to improve the efficiency of behavior analysis, thereby effectively solving the problem of low behavior analysis capabilities existing in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0040] Figure 1 It is a schematic diagram of the implementation environment involved in this application;

[0041] Figure 2 is a flow chart of a behavior analysis method according to an exemplary embodiment;

[0042] Figure 3 yes Figure 2 A flowchart of step 210 in one embodiment corresponding to the embodiment;

[0043] Figure 4 yes Figure 3 A flowchart of an embodiment corresponding to step 211 in an embodiment;

[0044] Figure 5 yes Figure 3 A flowchart of an embodiment corresponding to step 213 in an embodiment;

[0045] Figure 6 yes Figure 2 A flowchart of step 230 in one embodiment corresponding to the embodiment;

[0046] Figure 7 yes Figure 6 A flowchart of step 233 in one embodiment corresponding to the embodiment;

[0047] Figure 8 yes Figure 7 A flowchart of an embodiment corresponding to step 2331 in an embodiment;

[0048] Fig. 9 It is a specific implementation flow chart of a behavior analysis method in an application scenario;

[0049] Fig.10 is a structural block diagram of a behavior analysis device according to an exemplary embodiment;

[0050] Fig.11 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0051] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0052] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0053] As mentioned above, in the prior art, existing animal behavior analysis methods mainly extract interesting body point features from animal videos, and perform behavior analysis through methods including numerical calculation, behavior machine learning classification, clustering, etc. Among them, the existing methods perform behavior analysis through behavior machine learning classification or clustering, and the analysis accuracy is limited. At the same time, the existing methods focus on a single small-scale data and cannot extract general behavioral features. The experiment is difficult to reproduce. It is processed through multiple stages and takes a long time, resulting in low analysis efficiency.

[0054] To this end, the behavior analysis method provided in the present application can effectively improve the efficiency of behavior analysis. Accordingly, the behavior analysis method is applicable to a behavior analysis device, which can be deployed in an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a laptop computer, a server, etc.; the electronic device can also be an electronic device with a central control function, for example, the electronic device includes a gateway, etc.; the electronic device can also refer to a portable and mobile electronic device, for example, the electronic device includes a smart phone, a tablet computer, etc.

[0055] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0056] Figure 1 It is a schematic diagram of an implementation environment involved in a behavior analysis method. It should be noted that this implementation environment is only an example adapted to the present invention and cannot be considered as providing any limitation on the scope of use of the present invention.

[0057] The implementation environment includes a collection end 110 and a service end 130 .

[0058] Specifically, the acquisition end 110 can be considered as a data acquisition device, including but not limited to electronic devices with information acquisition functions such as recorders and cameras, and can also be considered as a device that integrates information acquisition and behavior analysis, including but not limited to electronic devices such as laptops and desktop computers that have both information acquisition and behavior analysis functions. That is, in some embodiments, the acquisition end 110 can collect multimodal data.

[0059] The server 130 can also be considered as a behavior analysis device, including but not limited to smart phones, desktop computers, laptop computers, servers and other electronic devices with behavior analysis functions, and can also be a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. The server 130 is used to provide background services, for example, the background services include but are not limited to behavior analysis services, etc.

[0060] The server 130 and the collection terminal 110 establish a network communication connection in advance through wired or wireless means, and the data transmission between the server 130 and the collection terminal 110 is realized through the network communication connection. The transmitted data includes but is not limited to: multimodal data and the like.

[0061] In an application scenario, through the interaction between the collection end 110 and the server end 130, the collection end 110 collects multimodal data and uploads the multimodal data to the server end 130 to request the server end 130 to provide behavior analysis services.

[0062] At this time, for the server 130, after receiving the multimodal data uploaded by the acquisition end 110, it calls the behavior analysis service, trains the multimodal large model according to the multimodal data, and generates a behavioral text description for the target to be detected, and then generates a behavioral analysis result for the target to be detected based on the large language model, which solves the problem of low efficiency of behavior analysis in related technologies.

[0063] See also Figure 2 The present application embodiment provides a behavior analysis method, which is applicable to an electronic device, which may be Figure 1 A server 130 in an implementation environment is shown.

[0064] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0065] like Figure 2 As shown, the method may include the following steps:

[0066] Step 210, obtain multimodal data collected for the target to be detected, and perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a multimodal large model.

[0067] Among them, multimodal data is data collected for the target to be detected and carries the behavioral status information of the target to be detected. Multimodal data can be data in a single form or a data set including multiple data. For example, multimodal data can be video data collected by shooting the target to be detected, or it can be EEG recording data, surrounding environment data, sound data, etc. collected for the target to be detected.

[0068] Step 230 , performing behavioral analysis on the multimodal data based on the multimodal large model to obtain a text description of the behavior of the target to be detected.

[0069] In the process of behavior analysis, it is necessary to format the acquired multimodal data into a text description, and then obtain a common behavior text description in text form as the basis for behavior analysis. In this application, various forms of multimodal data are unified and integrated into a textual behavior text description through behavior analysis.

[0070] Step 250 , statistically analyzing the behavior text description based on the large language model to obtain a behavior analysis result for the target to be detected.

[0071] Specifically, the behavior text descriptions generated by the multimodal large model are statistically summarized, and the behavior text descriptions are input into the large language model as prompt words, thereby guiding the large language model to output the behavior state information of the target to be detected related to the behavior text descriptions.

[0072] In a possible implementation, video data of the animal is collected, and the collected video data is analyzed to generate behavior analysis results including status information such as the animal's emotion or disease level.

[0073] Through the above process, this application can learn large-scale multimodal data by fine-tuning the pre-trained model with multimodal data, allowing the model to better utilize large-scale data, thereby realizing the extraction of general behavioral features and analysis, improving the accuracy of behavior classification, recognition, and description, and obtaining a multimodal large model with high behavior description accuracy, which can effectively improve the efficiency of behavior analysis. Through the joint use of multimodal large models and large language models, behavior analysis of multimodal data is performed in an end-to-end manner to improve the efficiency of behavior analysis.

[0074] In an exemplary embodiment, if Figure 3 As shown, step 210 may include the following steps:

[0075] Step 211, constructing a behavior semantic dataset based on multimodal data.

[0076] Among them, the semantic data set includes multimodal data segments with corresponding behavioral semantics.

[0077] In an exemplary embodiment, if Figure 4 As shown, step 211 may include the following steps:

[0078] Step 2111: perform cluster analysis on the multimodal data to divide the multimodal data into at least two short-term data segments.

[0079] The short-term data segments are segments obtained by dividing the multimodal data. The short-term data segments are obtained by dividing the multimodal data into different time lengths.

[0080] In one possible implementation, the multimodal data is two-dimensional or three-dimensional posture estimation data including body point data. At this time, cluster analysis is performed on the body points through tools such as moseq and Behavior Atlas to obtain short-term data segments corresponding to different clusters.

[0081] Step 2113, setting corresponding behavioral semantics for the short-term data segments to generate a behavioral semantic data set.

[0082] It can be understood that different short-term data segments can represent different behavioral processes of the target to be detected during the data collection process. Therefore, corresponding behavioral semantic labels can be set for the short-term data segments.

[0083] In one possible implementation, data is collected during the behavior of the target to be detected that has specific behavioral semantics, and the corresponding specific behavioral semantics are set by selecting short-term data segments in the collected multimodal data. For example, the running process of a mouse is filmed, and then the "running" behavioral semantics is set for the mouse running segment in the captured video data.

[0084] Through the above process, a behavioral semantic data set is generated through cluster analysis, which can indicate the model training direction in the subsequent training process, thereby improving the behavioral analysis efficiency and accuracy of the generated multimodal large model.

[0085] Step 213, conduct behavior analysis training on the pre-trained model based on the behavior semantic data set to obtain a multimodal large model.

[0086] In an exemplary embodiment, if Figure 5 As shown, step 213 may include the following steps:

[0087] Step 2131, input the behavior semantic data set into the pre-trained model to perform behavior analysis, and obtain a behavior text description for each short-term data segment in the behavior semantic data set.

[0088] Among them, the pre-trained model is a language model that generates corresponding behavioral text descriptions based on multimodal data. By inputting short-term data fragments in the behavioral semantic data set into the pre-trained model, the pre-trained model generates text for describing the behavioral semantics corresponding to the short-term data fragments by detecting the short-term data fragments.

[0089] For example, a video clip corresponding to a mouse running is input into a pre-trained model, and the pre-trained model generates a behavioral text description of "the mouse in the video is moving at an extremely fast speed, that is, running. Its speed is *** and its acceleration is ***."

[0090] Step 2133, based on the behavior semantics in the semantic data set, the accuracy of the behavior text description is evaluated to obtain an evaluation result indicating the accuracy of the behavior description of the behavior semantics of the short-term data segment by the pre-trained model.

[0091] Among them, by comparing the behavior text description generated by the pre-trained model with the pre-set behavior semantics, the behavior analysis ability of the pre-trained model is judged and an evaluation result is generated.

[0092] Step 2135, adjust the parameters of the pre-trained model based on the evaluation results and repeat the training until the behavior description accuracy of the pre-trained model behavior analysis reaches a preset threshold, and a multimodal large model is obtained.

[0093] In a possible implementation, the data in the semantic dataset is divided into a training set, a validation set, and a test set. The pre-trained model is trained with the training set, and the behavior description accuracy of the pre-trained model is calculated with the validation set and the test set, wherein the accuracy in the test set is obtained by evaluation by personnel in the field.

[0094] In the above process, the pre-trained model is finely trained through refined short-term data fragments to generate a target multimodal large model. Through supervised fine-tuning and alignment of the training model with behavioral semantics, the training model is trained to generate more accurate and detailed text descriptions of the behavior, and finally a target multimodal large model that can perform refined behavior analysis is obtained. It can achieve fine-tuning of the model locally on a large scale using high-quality and scientific data sets, improving training efficiency and the accuracy and efficiency of model behavior classification, recognition, and description.

[0095] In an exemplary embodiment, if Figure 6 As shown, step 230 may include the following steps:

[0096] Step 231, generating corresponding question information based on the multimodal data.

[0097] The question information is text generated for the type of multimodal data. By inputting the question information into the multimodal big model as prompt word information, the multimodal big model generates a text description of the behavior corresponding to the question information.

[0098] For example, when the multimodal information is mouse video information, “Please tell me what the mouse in the video is doing, and its specific motion parameters, such as speed and acceleration” is input into the multimodal large model as question information.

[0099] Step 233: Generate a behavior text description in a preset format corresponding to the multimodal data based on the question information.

[0100] The behavior text description for the multimodal data and each short-term data segment therein is standardized by a preset format. For example, the behavior text description for each short-term data segment is generated by a three-segment description, and the behavior text description for the entire multimodal data is generated by a two-segment description.

[0101] In a possible implementation, the behavior text description format for each short-term data segment is {detailed description, definition, motion parameters}, and the behavior text description format for the entire multimodal data is {behavior sequence, motion parameters}.

[0102] In an exemplary embodiment, if Figure 7 As shown, step 233 may include the following steps:

[0103] Step 2331, generating a corresponding behavior text description for each short-term data segment in the multimodal data.

[0104] In a possible implementation, the behavior text description may include behavior semantic text and motion parameter text.

[0105] In an exemplary embodiment, if Figure 8 As shown, step 2331 may include the following steps:

[0106] Step 23311, determine the three-dimensional body point data corresponding to the target to be detected in the short-time data segment.

[0107] Among them, the three-dimensional body point data is the data obtained through three-dimensional motion collection after setting fixed data points for the target to be detected. The three-dimensional body points can be used to quantify the specific motion parameters of the target to be detected during the detection process.

[0108] Step 23313, predict motion parameters based on three-dimensional body point data and generate motion parameter text.

[0109] Among them, the motion parameters are parameters calculated by obtaining the relative position and motion process of each data point in the three-dimensional body point data. For example, by performing behavioral semantic prediction on the mouse video, the motion parameter text of "the mouse speed is 10m / min, and the acceleration is 0" is generated.

[0110] In a possible implementation, the motion parameters are not limited to speed and acceleration, but also include parameters such as body part orientation, angle, and body length.

[0111] Step 23315, perform behavioral semantic prediction on the short-term data segment and generate behavioral semantic text.

[0112] Among them, semantic text is text information used to describe the behavior of the target to be detected in a short data segment. For example, by performing behavioral semantic prediction on a mouse video, a behavioral semantic text is generated, such as "the mouse in the video is licking his fur, combing it with his front paws, or scratching, that is, grooming his fur."

[0113] Through the above process, accurate behavioral semantic text can be achieved for short-term data fragments, thereby improving the accuracy of model behavior analysis.

[0114] Step 2333, generating a corresponding multimodal data behavior text description for the behavior sequence of each short-term data segment.

[0115] It should be noted that the short-term data segments in the multimodal data have a time sequence relationship, so the behavior text description of each short-term data segment is summarized through the behavior sequence to generate the corresponding multimodal data behavior text description.

[0116] For example, by performing behavioral semantic prediction on a mouse video, a behavioral semantic text is generated: "The mouse in the video is grooming in the first 10 seconds and sniffing in the last 5 seconds. Its movement distance in the video is 0 and its speed does not change."

[0117] Through the above process, a refined behavioral analysis of multimodal data can be achieved, and the behavioral semantic text of multimodal data can be accurately obtained, ensuring that the statistical analysis performed through the large language model can accurately obtain the behavioral information of the target to be detected, thereby improving the accuracy and efficiency of the behavioral analysis.

[0118] Fig. 9 This is a schematic diagram of a specific implementation of a behavior analysis method in an application scenario.

[0119] In this application scenario, a high-performance server with A100 80G is used to train large multimodal models and perform behavioral analysis.

[0120] Among them, Fig. 9 As shown, by obtaining mouse videos, collecting two-dimensional or three-dimensional posture estimation data, and using tools such as moseq and Behavior Atlas to perform cluster analysis on body points, based on the clustering results, short-term video clips with behavioral semantics are selected to generate a semantic dataset.

[0121] The pre-trained model is trained using semantic datasets to obtain a large multimodal model.

[0122] The question information “Please tell me what the mouse in the video is doing, and its specific movement parameters, such as speed and acceleration” is input into the multimodal model.

[0123] The multimodal large model generates a text description of the behavior for a short video clip according to the format of {detailed description, definition, motion parameters}: "The mouse in the video is licking his fur, combing it with his front paws, or scratching, that is, grooming his fur. His velocity is 0 and his acceleration is 0."

[0124] Among them, because videos that are too long or have too high a resolution cannot be processed, the model is used to loop through the mouse videos to obtain short-term and long-term video segments in the video.

[0125] The multimodal large model generates behavioral text descriptions for long video clips according to the format of {behavior sequence, motion parameters} - "The mouse in the video first grooms itself for 3 seconds, then starts running for 2 seconds, then turns to the right and continues jogging for 2 seconds, and finally stops to sniff for 2 seconds. The distance it moves in the video is ***, and its speed shows a trend of accelerating first and then decelerating."

[0126] The behavioral analysis results are generated by summarizing the behavioral text descriptions generated by the multimodal large model and inputting them into the large language model for data statistical analysis.

[0127] In the present invention, the use of a multimodal large model for refined behavior analysis of animals is realized. A method based on a multimodal large model is proposed to perform behavior analysis on animal behavior video data in an end-to-end manner. Based on a pre-trained multimodal large model, the model is fine-tuned locally using a high-quality, scientific behavior video dataset to obtain better refined behavior analysis results than the existing technology. The model structure and parameter scale of the multimodal large model allow the model to better utilize large-scale data, making it possible to extract general behavioral features and analysis. A large language model is used to perform statistical analysis on the behavior text description output by the complete video to improve the efficiency of behavior analysis.

[0128] The following is an embodiment of the device of the present application, which can be used to perform the behavior analysis method involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the behavior analysis method involved in the present application.

[0129] See also Fig.10 In an embodiment of the present application, a behavior analysis device 900 is provided, including but not limited to: a data acquisition module 910, a behavior analysis module 930, and a statistical analysis module 950.

[0130] Among them, the data acquisition module 910 is used to obtain multimodal data collected for the target to be detected, and perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a multimodal large model.

[0131] The behavior analysis module 930 is used to perform behavior analysis on the multimodal data based on the multimodal large model to obtain a text description of the behavior of the target to be detected.

[0132] The statistical analysis module 950 is used to perform statistical analysis on the behavior text description based on the large language model to obtain the behavior analysis result for the target to be detected.

[0133] It should be noted that the behavior analysis device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing behavior analysis. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the behavior analysis device will be divided into different functional modules to complete all or part of the functions described above.

[0134] In addition, the behavior analysis device and behavior analysis method embodiments provided in the above embodiments belong to the same concept, wherein the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0135] See also Fig.11 In an embodiment of the present application, an electronic device 4000 is provided, and the electronic device 400 may include: a desktop computer, a laptop computer, a server, etc.

[0136] exist Fig.11 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0137] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0138] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0139] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0140] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to this.

[0141] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0142] The computer-readable instructions are executed by one or more processors 4001 to implement the behavior analysis method in the above-mentioned embodiments.

[0143] In addition, an embodiment of the present application provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the above behavior analysis method.

[0144] A computer program product is provided in an embodiment of the present application. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the above-mentioned behavior analysis method.

[0145] Compared with the related art, the present application can learn large-scale multimodal data by fine-tuning the pre-trained model with multimodal data, allowing the model to make better use of large-scale data, thereby realizing the extraction of general behavioral features and analysis, improving the accuracy of behavior classification, recognition, and description, and obtaining a multimodal large model with high behavior description accuracy, which can effectively improve the efficiency of behavior analysis. Through the joint use of multimodal large models and large language models, behavior analysis is performed on multimodal data in an end-to-end manner to improve the efficiency of behavior analysis. The generation of behavioral semantic data sets is achieved through cluster analysis, so that the model training direction can be indicated in the subsequent training process, thereby improving the efficiency and accuracy of behavior analysis of the generated multimodal large model. The pre-trained model is fine-tuned through refined short-term data fragments to generate a target multimodal large model. Through the behavior semantics, the model to be trained is supervised fine-tuned and aligned, so that the model to be trained is trained in the direction of generating more accurate and more refined behavior text descriptions, and finally the target multimodal large model that can perform refined behavior analysis is obtained, which can realize the fine-tuning of the model using high-quality and scientific data sets on a large scale locally, improving the training efficiency and the accuracy and efficiency of model behavior classification, recognition, and description. By realizing refined behavioral analysis of multimodal data, the behavioral semantic text of multimodal data can be accurately obtained, ensuring that the statistical analysis performed through the large language model can accurately obtain the behavioral information of the target to be detected, thereby improving the accuracy and efficiency of behavioral analysis.

[0146] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0147] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A behavior analysis method, characterized in that: The method comprises: Obtain multimodal data collected for the target to be detected, and perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a multimodal large model; Performing behavioral analysis on the multimodal data based on a multimodal large model to obtain a behavioral text description of the target to be detected; The behavior text description is statistically analyzed based on a large language model to obtain a behavior analysis result for the target to be detected.

2. The method according to claim 1, characterized in that The performing behavior analysis training on the pre-trained model based on the multimodal data to obtain a multimodal large model includes: Constructing a behavioral semantic dataset based on the multimodal data; Based on the behavioral semantic data set, the pre-trained model is trained for behavior analysis to obtain a multimodal large model.

3. The method according to claim 2, characterized in that The constructing a behavior semantic data set based on the multimodal data includes: Performing cluster analysis on the multimodal data to divide the multimodal data into at least two short-term data segments; Corresponding behavioral semantics are set for the short-term data segments to generate a behavioral semantic data set.

4. The method according to claim 2, characterized in that The method of performing behavior analysis training on the pre-trained model based on the behavior semantic data set to obtain a multimodal large model includes: Inputting the behavior semantic data set into the pre-trained model to perform behavior analysis, and obtaining a behavior text description for each short-term data segment in the behavior semantic data set; Evaluate the accuracy of the behavior text description based on the behavior semantics in the semantic data set to obtain an evaluation result indicating the accuracy of the behavior description of the behavior semantics of the short-term data segment by the pre-trained model; Based on the evaluation result, the parameters of the pre-trained model are adjusted and the training is repeated until the behavior description accuracy of the behavior analysis of the pre-trained model reaches a preset threshold, thereby obtaining a large multimodal model.

5. The method according to claim 1, characterized in that The performing behavior analysis on the multimodal data based on the multimodal large model to obtain a behavior text description of the target to be detected includes: generating corresponding question information based on the multimodal data; A behavior text description in a preset format corresponding to the multimodal data is generated based on the question information.

6. The method according to claim 5, characterized in that The generating a text description of the behavior corresponding to the multimodal data based on the question information includes: Generating a corresponding behavior text description for each short-term data segment in the multimodal data; A text description of the multimodal data behavior is generated for the behavior sequence of each of the short-term data segments.

7. The method according to claim 6, characterized in that The behavior text description includes behavior semantic text and motion parameter text; The generating a corresponding behavior text description for each short-term data segment in the multimodal data includes: Determining three-dimensional body point data corresponding to the target to be detected in the short-term data segment; Predicting motion parameters based on the three-dimensional body point data and generating motion parameter text; Perform behavioral semantic prediction on the short-term data segment to generate behavioral semantic text.

8. A behavior analysis device, characterized in that: include: A data acquisition module is used to obtain multimodal data collected for the target to be detected, and to perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a multimodal large model; A behavior analysis module, used to perform behavior analysis on the multimodal data based on a multimodal large model to obtain a text description of the behavior of the target to be detected; The statistical analysis module is used to perform statistical analysis on the behavior text description based on a large language model to obtain a behavior analysis result for the target to be detected.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors so that the electronic device implements the behavior analysis method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the behavior analysis method according to any one of claims 1 to 7.

Images