Behavior recognition method, system, device and storage medium based on scene understanding

Abstract

The present invention belongs to the field of computer vision technology and provides a method, system, device, and storage medium for behavior recognition based on scene understanding, aiming to solve the technical problems existing in the prior art, such as low accuracy, susceptibility to interference, inability to effectively guarantee precision, and inability to effectively support behavior recognition. The method comprises: obtaining an image frame including a target captured by a video surveillance device; performing target recognition processing on the image frame using a target recognition model to obtain target entity information of the image frame; performing entity attribute relationship analysis on the target entity information to obtain target attribute information, and constructing a unified behavior recognition representation vector for the target based on the target attribute information; calculating the unified behavior recognition representation vector using preset weights to obtain a state change degree value; and using the state change degree value as the behavior recognition result.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a behavior recognition method, system, device and storage medium based on scene understanding. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, scene understanding technology has been widely used in intelligent monitoring, autonomous driving, human-computer interaction and other fields.

[0003] Existing behavior recognition methods mainly include recognition methods including region selection (sliding window), feature extraction (SIFT, HOG, etc.), and classifiers (SVM, Adaboost, etc.); behavior recognition methods based on deep learning; models such as CLIP, GLIP, and BLIP; and behavior recognition methods based on generative multimodal large models.

[0004] Existing behavior recognition methods have the following technical problems:

[0005] Existing detection methods have low accuracy, are susceptible to interference, cannot effectively guarantee precision, and cannot effectively support behavior recognition. Summary of the Invention

[0006] The present invention provides a behavior recognition method, system, device and storage medium based on scene understanding, aiming to solve the technical problems existing in the above-mentioned existing technologies, such as low accuracy, susceptibility to interference, ineffective precision guarantee, and inability to effectively support behavior recognition.

[0007] The present invention solves the above technical problems with the following technical solutions: A behavior recognition method based on scene understanding, comprising:

[0008] Acquire an image frame including a target captured by a video surveillance device;

[0009] Performing target recognition processing on the image frame through a target recognition model to obtain target entity information of the image frame;

[0010] Performing entity attribute relationship analysis on the target entity information to obtain target attribute values, and constructing a unified behavior recognition representation vector for the target based on the target attribute values;

[0011] The behavior recognition unified representation vector is calculated using the preset weights to obtain the state change degree value.

[0012] The state change degree value is used as the behavior recognition result.

[0013] Furthermore, the target recognition model includes a discriminative multimodal large model or a dedicated small model, and performing target recognition processing on the image frame by using the discriminative multimodal large model specifically includes:

[0014] Obtaining an initial prompt word corresponding to the image frame, and preprocessing the initial prompt word to obtain a target prompt word;

[0015] The image frame and the target prompt word are input into the discriminative multimodal large model, and the discriminative multimodal large model is used to perform target recognition processing on the image frame to obtain the target entity information.

[0016] Furthermore, the target recognition model includes a discriminative multimodal large model and a dedicated small model, and obtaining target entity information of the image frame specifically includes:

[0017] Inputting the image frame into the discriminative multimodal large model, and inputting the preset target prompt word into the discriminative multimodal large model;

[0018] Using the discriminative multimodal large model to perform target recognition processing on the image frame according to the target prompt word to obtain initial entity information;

[0019] Inputting the initial entity information and the image frame into a dedicated small model;

[0020] Based on the initial entity information, using the dedicated small model to perform image segmentation on the image frame to obtain a segmented image;

[0021] The segmented image is refined for recognition to obtain target entity information.

[0022] Furthermore, the above-mentioned construction of the discriminant multimodal large model specifically includes:

[0023] Construct an initial multimodal large model;

[0024] Get a dataset D of multiple categories i , for all the datasets D i Preprocess and merge to obtain the comprehensive dataset D all ; where i = 1, 2, ... n;

[0025] Using various preprocessed data sets D i The initial multimodal large model is trained to obtain an updated multimodal large model M i ;

[0026] Using the comprehensive dataset D all Update the multimodal large model M n Train and get the updated multimodal large model M all ;

[0027] Update the obtained multimodal large model M i and update the multimodal large model Mall Perform performance evaluation and adjust each updated multimodal large model M based on the performance evaluation results i Training parameters and updating of the multimodal large model M all After the training parameters are adjusted, the multimodal large model M is updated in a loop. i The training steps and the use of the comprehensive dataset D all Update the multimodal large model M n Train and get the updated multimodal large model M all Training steps until the multimodal large model M is updated all The performance evaluation result reaches the preset evaluation threshold, or the multimodal large model M is updated all When the performance of the training is stable, stop training;

[0028] The updated multimodal large model M obtained from the last training all As the discriminative multimodal large model;

[0029] Among them, update the multimodal large model M i The training steps include:

[0030] The initial multimodal large model is trained using the preprocessed data set D1 to obtain an updated multimodal large model M1, and the updated multimodal large model M1 is trained using the preprocessed data set D2 to obtain an updated multimodal large model M2, and so on until the updated multimodal large model M1 is trained using the preprocessed data set D n Update the multimodal large model M n-1 Train and get the updated multimodal large model M n .

[0031] Furthermore, the above-mentioned construction of a unified target behavior recognition representation vector based on the target attribute value specifically includes:

[0032] Performing entity attribute relationship analysis on the target entity information to obtain target attribute values; wherein the target entity information includes: category information, and part or all of four types of information: quantity, area, location, and color;

[0033] Based on the target attribute value, a unified behavior recognition representation vector for targets of each category is constructed according to the category information; wherein the unified behavior recognition representation vector includes the target attribute value of other entity attribute information in the entity attribute information except the category information and the relative increment of the target attribute value.

[0034] Furthermore, the above-mentioned state change degree value specifically includes:

[0035] Assigning a preset weight value to each element in the unified behavior recognition representation vector to obtain a weight vector;

[0036] Performing L1 norm calculation and L2 norm calculation on the weight vector respectively to obtain a first norm value and a second norm value;

[0037] The first norm value and the second norm value are weighted based on a preset weight to obtain a state change degree value.

[0038] Furthermore, the above-mentioned state change degree value specifically includes:

[0039] Extracting elements from the unified behavior recognition representation vector, and constructing a subvector of the unified behavior recognition representation vector based on the extracted elements;

[0040] Assigning a preset weight value to each element in the sub-vector to obtain a sub-weight vector;

[0041] Performing L1 norm calculation and L2 norm calculation on the sub-weight vector respectively to obtain a first sub-norm value and a second sub-norm value;

[0042] The first sub-norm value and the second sub-norm value are weighted based on a preset weight to obtain a state change degree value.

[0043] In a second aspect, in order to solve the above technical problems, the present invention further provides a behavior recognition system based on scene understanding, comprising:

[0044] A data acquisition module is used to acquire image frames including the target captured by the video surveillance equipment;

[0045] A data recognition module is used to perform target recognition processing on the image frame through a target recognition model to obtain target entity information of the image frame;

[0046] A behavior characterization module is used to perform entity attribute relationship analysis on the target entity information to obtain target attribute values, and to construct a unified behavior recognition characterization vector for the target based on the target attribute values;

[0047] The behavior recognition module is used to calculate the behavior recognition unified representation vector through preset weights to obtain a state change degree value, and use the state change degree value as the behavior recognition result.

[0048] In the third aspect, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the behavior recognition method based on scene understanding of the present application is implemented.

[0049] In a fourth aspect, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the behavior recognition method based on scene understanding of the present application is implemented.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] 1. By using a discriminative multimodal large model, this invention can effectively process information from different data sources and improve the accuracy of target recognition. It can fully utilize the advantages of multimodal information and provide more comprehensive target recognition results.

[0052] 2. The present invention adopts a step-by-step iterative approach to train the initial multimodal large model and adjusts parameters at each step according to the performance evaluation results to ensure that the final discriminant multimodal large model has high accuracy and robustness.

[0053] 3. This invention constructs a comprehensive, unified representation vector for behavior recognition by comprehensively considering multiple dimensions of target entity information, including category, quantity, area, location, and color. This not only considers the target's basic attributes but also its relative increments, providing a rich information foundation for subsequent calculations of the degree of state change.

[0054] 4. This invention utilizes a norm calculation method to assign different weights to the elements of the unified behavior recognition representation vector and calculates the degree of state change based on this weight, enabling a more accurate reflection of changes in the target behavior. Furthermore, subvectors are extracted for detailed analysis, enhancing the flexibility and adaptability of the entire system.

[0055] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 A schematic flow chart of a behavior recognition method based on scene understanding according to an embodiment of the present invention is shown;

[0058] Figure 2 A flow chart of a behavior recognition method based on scene understanding according to an embodiment of the present invention is shown;

[0059] Figure 3 A schematic diagram of a scenario of a behavior recognition method based on scenario understanding according to an embodiment of the present invention is shown;

[0060] Figure 4 A schematic structural diagram of a behavior recognition system based on scene understanding according to an embodiment of the present invention is shown;

[0061] Figure 5 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0063] Figure 1 FIG. 4 shows a flow chart of a behavior recognition method based on scene understanding according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, a behavior recognition method based on scene understanding in this embodiment includes:

[0064] Acquire an image frame including a target captured by a video surveillance device;

[0065] Performing target recognition processing on the image frame through a target recognition model to obtain target entity information of the image frame;

[0066] Performing entity attribute relationship analysis on the target entity information to obtain target attribute values, and constructing a unified behavior recognition representation vector for the target based on the target attribute values;

[0067] Calculating the behavior recognition unified representation vector using a preset weight to obtain a state change degree value;

[0068] The state change degree value is used as the behavior recognition result.

[0069] In summary, this embodiment processes image frames using a target recognition model to more accurately identify target entities within the image. By acquiring target attribute information and values, it maintains high recognition accuracy even in complex backgrounds or in the presence of interference, improving the interference resistance of behavior recognition. By calculating the degree of state change, it ensures the accuracy of behavior recognition.

[0070] like Figure 2 As shown, in this embodiment, whether the state change degree value reaches the alarm threshold is determined based on the state change degree value, and an alarm is triggered if the alarm threshold is reached.

[0071] like Figure 3 As shown, it is a schematic diagram of the scenario of this embodiment. The scenario is composed of video surveillance facilities, networks, servers, and clients. Among them, the scenes and behaviors in the incident area include but are not limited to mechanical construction scenes and behaviors, fishing scenes and behaviors, illegal parking scenes and behaviors of motor vehicles or non-motor vehicles, warehouse fire scenes, road safety hazard scenes, etc.; the video surveillance facilities are mainly visible light dome cameras, visible light gun cameras, and brackets or bases for installing cameras, ranging equipment, etc.

[0072] Optionally, the target recognition model includes a discriminative multimodal large model or a dedicated small model, and performing target recognition processing on the image frame using the discriminative multimodal large model specifically includes:

[0073] Obtaining an initial prompt word corresponding to the image frame, and preprocessing the initial prompt word to obtain a target prompt word;

[0074] The image frame and the target prompt word are input into the discriminative multimodal large model, and the discriminative multimodal large model is used to perform target recognition processing on the image frame to obtain the target entity information.

[0075] In this embodiment, the discriminant multimodal large model adopts the Grounding DINO model.

[0076] In this embodiment, the prompt words include: "Detect the main target in the image and describe this image." or "Describe this image." or "Give a description of the scene in this image." or "Describe the behavior in this image." or "Describe the scene in this image and analyze the behavior in this image." Regarding the prompt words, if the initial prompt words input are Chinese sentences, the corresponding target recognition model used should be the Chinese kernel model; if the target recognition model used is the English kernel model, the initial prompt words need to be preprocessed and translated into English prompt words. Conversely, if the target recognition model used is the Chinese kernel model, the initial prompt words in English need to be preprocessed and translated into Chinese prompt words.

[0077] Optionally, the target recognition model includes a discriminative multimodal large model and a dedicated small model, and obtaining target entity information of the image frame specifically includes:

[0078] Inputting the image frame into the discriminative multimodal large model, and inputting the preset target prompt word into the discriminative multimodal large model;

[0079] Using the discriminative multimodal large model to perform target recognition processing on the image frame according to the target prompt word to obtain initial entity information;

[0080] Inputting the initial entity information and the image frame into a dedicated small model;

[0081] Based on the initial entity information, using the dedicated small model to perform image segmentation on the image frame to obtain a segmented image;

[0082] The segmented image is refined for recognition to obtain target entity information.

[0083] In this embodiment, when faster response and reduced resource usage are required (i.e., when time consumption and GPU or CPU resources are more sensitive, while accuracy requirements are lower), the discriminative multimodal large model stage can be skipped, and the target entity information output by the dedicated small model can be used directly as input for subsequent steps. Alternatively, the dedicated small model stage can be skipped, and the target entity information output by the discriminative multimodal large model can be used directly as input for subsequent steps. When optimal accuracy is required, the discriminative multimodal large model and the dedicated small model are combined for processing.

[0084] Optionally, constructing the discriminant multimodal large model specifically includes:

[0085] Construct an initial multimodal large model;

[0086] Get a dataset D of multiple categories i , for all the datasets D i Preprocess and merge to obtain the comprehensive dataset D all ; where i = 1, 2, ... n;

[0087] Using various preprocessed data sets D i The initial multimodal large model is trained to obtain an updated multimodal large model M i ;

[0088] Using the comprehensive dataset D all Update the multimodal large model M n Train and get the updated multimodal large model M all ;

[0089] Update the obtained multimodal large model M iand update the multimodal large model M all Perform performance evaluation and adjust each updated multimodal large model M based on the performance evaluation results i Training parameters and updating of the multimodal large model M all After the training parameters are adjusted, the multimodal large model M is updated in a loop. i The training steps and the use of the comprehensive dataset D all Update the multimodal large model M n Train and get the updated multimodal large model M all Training steps until the multimodal large model M is updated all The performance evaluation result reaches the preset evaluation threshold, or the multimodal large model M is updated all When the performance of the training is stable, stop training;

[0090] The updated multimodal large model M obtained from the last training all As the discriminant multimodal large model; wherein, the updated multimodal large model M i The training steps include:

[0091] The initial multimodal large model is trained using the preprocessed data set D1 to obtain an updated multimodal large model M1, and the updated multimodal large model M1 is trained using the preprocessed data set D2 to obtain an updated multimodal large model M2, and so on until the updated multimodal large model M1 is trained using the preprocessed data set D n Update the multimodal large model M n-1 Train and get the updated multimodal large model M n .

[0092] Optionally, constructing a unified target behavior recognition representation vector based on the target attribute value specifically includes:

[0093] Performing entity attribute relationship analysis on the target entity information to obtain target attribute values; wherein the target entity information includes: category information, and part or all of four types of information: quantity, area, location, and color;

[0094] Based on the target attribute value, a unified behavior recognition representation vector for targets of each category is constructed according to the category information; wherein the unified behavior recognition representation vector includes the target attribute value of other entity attribute information in the entity attribute information except the category information and the relative increment of the target attribute value.

[0095] In this embodiment, the unified characterization vector for behavior recognition is: U (N, S, X, Y, R, G, B, fN, fS, fX, fY, fR, fG, fB);

[0096] Where N represents the number, S represents the area, X and Y represent the pixel horizontal and vertical coordinates of the target center, respectively. R, G, and B represent the target's R, G, and B color values, respectively. fN represents the relative increment of the number, fS represents the relative increment of the area, fX represents the relative increment of the pixel horizontal coordinate, fY represents the relative increment of the pixel vertical coordinate, fR represents the relative increment of the R color value, fG represents the relative increment of the G color value, and fB represents the relative increment of the B color value. There is no restriction on the target attribute value being represented by RGB color, nor is there a restriction on the target attribute value being represented by the horizontal and vertical coordinates.

[0097] In this embodiment, the relative increment of N is calculated as follows: (the number of a certain type of target in the current frame - the number of this type of target in the previous frame) / the number of this type of target in the previous frame;

[0098] S relative increment calculation method: (area of a target ID in the current frame - area of the target ID in the previous frame) / area of the target ID in the previous frame;

[0099] X relative increment calculation method: X relative increment = (pixel coordinate X of a target ID in the current frame - pixel coordinate X of the target ID in the previous frame) / pixel coordinate X of the target ID in the previous frame;

[0100] Y relative increment calculation method: Y relative increment = (pixel coordinate Y of a target ID in the current frame - pixel coordinate Y of the target ID in the previous frame) / pixel coordinate Y of the target ID in the previous frame;

[0101] R relative increment calculation method: R relative increment = (R channel value of a target ID in the current frame - R channel value of the target ID in the previous frame) / R channel value of the target ID in the previous frame;

[0102] G relative increment calculation method: G relative increment = (G channel value of a target ID in the current frame - G channel value of the target ID in the previous frame) / G channel value of the target ID in the previous frame;

[0103] B relative increment calculation method: B relative increment = (B channel value of a target ID in the current frame - B channel value of the target ID in the previous frame) / B channel value of the target ID in the previous frame;

[0104] If only one frame of image is used for algorithm analysis, the relative increment of the unified representation vector U for behavior recognition is 0, that is, U is (N, S, X, Y, R, G, B, 0, 0, 0, 0, 0, 0);

[0105] In this embodiment, different scenes have different characteristics, and based on these characteristics, scene descriptions can be performed. For example, in a mechanical construction scene, one or more mechanical vehicles such as excavators, forklifts, concrete trucks, and pump trucks may appear. Therefore, when the number of a certain mechanical vehicle in the unified behavior recognition representation vector is greater than or equal to 1, a description such as "how many mechanical vehicles of a certain type are there in the picture, this is a mechanical construction scene" can be given; based on the position coordinates X and Y in the unified behavior recognition representation vector, the positional relationship between different objects in a picture can be calculated, thereby describing the positional relationship between objects with different IDs of the same category or different categories;

[0106] Based on fN, fS, fX, fY, fR, fG, and fB in U, we can compare the changes in a target ID in different images (for example, images of the same scene, the same camera, the same configuration parameters, but at different times), so as to accurately determine its behavior (behavior in the ordinary sense (such as mechanical vehicle operation, motor vehicle driving in the wrong direction, etc.), or behavior in a broad sense (such as a change in land status from cultivated land to grassland)). For example, when the target is a person, a fishing rod, a plastic bucket, etc., the behavior recognition unified representation vector can be used to determine illegal fishing behavior; with the help of the descriptions such as quantity, area, relative increment of quantity, and relative increment of area in the behavior recognition unified representation vector, scene anomalies such as target loss or target damage can be determined; with the help of the descriptions such as quantity, area, color, relative increment of quantity, relative increment of area, and relative increment of RGB color in the behavior recognition unified representation vector, scene situations such as land abandonment can be determined.

[0107] Optionally, obtaining the state change degree value specifically includes:

[0108] Assigning a preset weight value to each element in the unified behavior recognition representation vector to obtain a weight vector;

[0109] Performing L1 norm calculation and L2 norm calculation on the weight vector respectively to obtain a first norm value and a second norm value;

[0110] The first norm value and the second norm value are weighted based on a preset weight to obtain a state change degree value.

[0111] In this embodiment, the preset weight values assigned to each element in the unified behavior recognition representation vector are in the range of [0, 1], where the weight values of N and S are 0.04, the weight values of X and Y are 0.05, the weight values of R, G, and B are 0.06, and the weight values of fN, fS, fX, fY, fR, fG, and fB are 0.1.

[0112] In this embodiment, the calculation formula for the state change degree value is:

[0113] val_last=w1×val_1+w2×val_2

[0114] Where val_1 represents the first norm value, val_2 represents the second norm value, and w1 and w2 represent weights, ranging from [0 to 1]. For example, w1 can be 0.4, and w2 can be 0.6.

[0115] Optionally, obtain the state change degree value, including:

[0116] Extracting elements from the unified behavior recognition representation vector, and constructing a subvector of the unified behavior recognition representation vector based on the extracted elements;

[0117] Assigning a preset weight value to each element in the sub-vector to obtain a sub-weight vector;

[0118] Performing L1 norm calculation and L2 norm calculation on the sub-weight vector respectively to obtain a first sub-norm value and a second sub-norm value;

[0119] The first sub-norm value and the second sub-norm value are weighted based on a preset weight to obtain a state change degree value.

[0120] In this embodiment, in order to perform scene understanding or behavior recognition more specifically, a sub-vector is used to calculate the state change degree value, for example: a sub-vector (N, S), or a sub-vector (X, Y, R, G, B), or a sub-vector (fS, fX, fY, fR, fG, fB) is used to assign a weight value, and finally the state change degree value is calculated.

[0121] Based on Figure 1 Based on the same principle as the method shown in , the embodiment of the present invention also provides a behavior recognition system based on scene understanding, such as Figure 4 As shown in , including:

[0122] A data acquisition module is used to acquire image frames including the target collected by the video surveillance equipment;

[0123] A data recognition module is used to perform target recognition processing on the image frame through a target recognition model to obtain target entity information of the image frame;

[0124] A behavior characterization module is used to perform entity attribute relationship analysis on the target entity information to obtain target attribute values, and to construct a unified behavior recognition characterization vector for the target based on the target attribute values;

[0125] The behavior recognition module is used to calculate the behavior recognition unified representation vector through preset weights to obtain a state change degree value, and use the state change degree value as the behavior recognition result.

[0126] The scene understanding-based behavior recognition system of the embodiment of the present invention can execute the scene understanding-based behavior recognition method provided by the embodiment of the present invention, and its implementation principle is similar. The actions performed by each module and unit in the scene understanding-based behavior recognition system in each embodiment of the present invention correspond to the steps in the scene understanding-based behavior recognition method in each embodiment of the present invention. For the detailed functional description of each module of the scene understanding-based behavior recognition system, please refer to the description of the corresponding scene understanding-based behavior recognition method shown in the previous text, and will not be repeated here.

[0127] Among them, the above-mentioned scene understanding-based behavior recognition system can be a computer program (including program code) running in a computer device, for example, the scene understanding-based behavior recognition system is an application software; the application software can be used to execute the corresponding steps in the method provided in the embodiment of the present invention.

[0128] In some embodiments, the behavior recognition system based on scene understanding provided by the embodiments of the present invention can be implemented by a combination of software and hardware. As an example, the behavior recognition system based on scene understanding provided by the embodiments of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the behavior recognition method based on scene understanding provided by the embodiments of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0129] The modules involved in the embodiments of the present invention may be implemented in software or hardware, wherein the name of a module does not necessarily limit the module itself.

[0130] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store computer programs; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0131] In an alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The electronic device shown includes a processor and a memory. The processor and the memory are connected, for example, via a bus. Optionally, the electronic device may further include a transceiver, which can be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, there is not limited to one transceiver, and the structure of the electronic device does not constitute a limitation on the embodiments of the present invention.

[0132] The processor may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0133] The bus may include a path for transmitting information between the above components. The bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0134] The memory may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0135] The memory is used to store application code (computer program) for executing the solution of the present invention, and the processor controls the execution of the application code. The processor is used to execute the application code stored in the memory to implement the content shown in the above method embodiment.

[0136] Among them, the electronic device can also be a terminal device, Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0137] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0138] According to another aspect of the present invention, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various implementations described above.

[0139] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0140] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0141] The computer-readable storage medium provided by the embodiments of the present invention may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0142] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0143] The above description is merely a preferred embodiment of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present invention.

Claims

1. A behavior recognition method based on scene understanding, characterized in that: The method comprises: Acquire an image frame including a target captured by a video surveillance device; Performing target recognition processing on the image frame through a target recognition model to obtain target entity information of the image frame; Performing entity attribute relationship analysis on the target entity information to obtain target attribute information; Constructing a unified characterization vector for target behavior recognition based on the target attribute information; Calculating the unified characterization vector for behavior recognition using a preset weight to obtain a state change degree value, and using the state change degree value as a behavior recognition result; The step of constructing a unified target behavior recognition representation vector based on the target attribute information specifically includes: Performing entity attribute relationship analysis on the target entity information to obtain target attribute values; wherein the target entity information includes: category information, and part or all of four types of information: quantity, area, location, and color; Based on the target attribute values, constructing a unified behavior recognition representation vector for each category of targets according to the category information; wherein the unified behavior recognition representation vector includes the target attribute values of other entity attribute information in the entity attribute information except the category information and the relative increment of the target attribute value; The state change degree value is obtained, specifically including: Assigning a preset weight value to each element in the unified behavior recognition representation vector to obtain a weight vector; Performing L1 norm calculation and L2 norm calculation on the weight vector respectively to obtain a first norm value and a second norm value; The first norm value and the second norm value are weighted based on a preset weight to obtain a state change degree value.

2. The behavior recognition method based on scene understanding according to claim 1 is characterized in that: The target recognition model includes a discriminative multimodal large model or a dedicated small model; Performing target recognition processing on the image frame using a discriminative multimodal large model, specifically including: Obtaining an initial prompt word corresponding to the image frame, and preprocessing the initial prompt word to obtain a target prompt word; The image frame and the target prompt word are input into the discriminative multimodal large model, and the discriminative multimodal large model is used to perform target recognition processing on the image frame to obtain the target entity information.

3. The behavior recognition method based on scene understanding according to claim 1 is characterized in that: The target recognition model includes a discriminative multimodal large model and a dedicated small model; Get the target entity information of the image frame, including: Inputting the image frame into the discriminative multimodal large model, and inputting the preset target prompt word into the discriminative multimodal large model; Using the discriminative multimodal large model to perform target recognition processing on the image frame according to the target prompt word to obtain initial entity information; Inputting the initial entity information and the image frame into a dedicated small model; Based on the initial entity information, using the dedicated small model to perform image segmentation on the image frame to obtain a segmented image; The segmented image is refined for recognition to obtain target entity information.

4. A behavior recognition method based on scene understanding according to claim 2 or 3, characterized in that: Constructing the discriminative multimodal large model specifically includes: Construct an initial multimodal large model; Get a dataset D of multiple categories i , for all the datasets D i Preprocess and merge to obtain the comprehensive dataset D all ; where i = 1, 2, ... n; Using various preprocessed data sets D i The initial multimodal large model is trained to obtain an updated multimodal large model M i ; Using the comprehensive dataset D all Update the multimodal large model M n Train and get the updated multimodal large model M all ; Update the obtained multimodal large model M i and update the multimodal large model M all Perform performance evaluation and adjust each updated multimodal large model M based on the performance evaluation results i Training parameters and updating of the multimodal large model M all After the training parameters are adjusted, the multimodal large model M is updated in a loop. i The training steps and the use of the comprehensive dataset D all Update the multimodal large model M n Train and get the updated multimodal large model M all Training steps until the multimodal large model M is updated all The performance evaluation result reaches the preset evaluation threshold, or the multimodal large model M is updated all When the performance of the training is stable, stop training; The updated multimodal large model M obtained from the last training all As the discriminant multimodal large model; wherein, the updated multimodal large model M i The training steps include: The initial multimodal large model is trained using the preprocessed data set D1 to obtain an updated multimodal large model M1, and the updated multimodal large model M1 is trained using the preprocessed data set D2 to obtain an updated multimodal large model M2, and so on until the updated multimodal large model M1 is trained using the preprocessed data set D n Update the multimodal large model M n-1 Train and get the updated multimodal large model M n .

5. The behavior recognition method based on scene understanding according to claim 1, characterized in that: Get the state change degree value, including: Extracting elements from the unified behavior recognition representation vector, and constructing a subvector of the unified behavior recognition representation vector based on the extracted elements; Assigning a preset weight value to each element in the sub-vector to obtain a sub-weight vector; Performing L1 norm calculation and L2 norm calculation on the sub-weight vector respectively to obtain a first sub-norm value and a second sub-norm value; The first sub-norm value and the second sub-norm value are weighted based on a preset weight to obtain a state change degree value.

6. A behavior recognition system based on scene understanding, characterized in that: include: A data acquisition module is used to acquire image frames including the target captured by the video surveillance equipment; A data recognition module is used to perform target recognition processing on the image frame through a target recognition model to obtain target entity information of the image frame; A behavior characterization module is used to perform entity attribute relationship analysis on the target entity information to obtain target attribute information, and to construct a unified behavior recognition characterization vector for the target based on the target attribute information; Specifically: Performing entity attribute relationship analysis on the target entity information to obtain target attribute values; wherein the target entity information includes: category information, and part or all of four types of information: quantity, area, location, and color; Based on the target attribute values, constructing a unified behavior recognition representation vector for each category of targets according to the category information; wherein the unified behavior recognition representation vector includes the target attribute values of other entity attribute information in the entity attribute information except the category information and the relative increment of the target attribute value; The behavior recognition module is used to calculate the behavior recognition unified representation vector using preset weights to obtain a state change degree value, and use the state change degree value as the behavior recognition result; specifically: Assigning a preset weight value to each element in the unified behavior recognition representation vector to obtain a weight vector; Performing L1 norm calculation and L2 norm calculation on the weight vector respectively to obtain a first norm value and a second norm value; The first norm value and the second norm value are weighted based on a preset weight to obtain a state change degree value.

7. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images