Multi-modal behavior analysis method and device, equipment and storage medium

Abstract

The invention relates to the field of image processing, and provides a multi-modal behavior analysis method and device, equipment and a storage medium. The method comprises the steps that an image to be analyzed is cut to obtain a plurality of sub-images to be analyzed, the detection effect is guaranteed through a relatively small image scale, interference of a large area without a detection object is eliminated, and time consumption and the effect are balanced; performing feature extraction on the plurality of to-be-analyzed sub-graphs to obtain a plurality of local feature graphs, respectively obtaining respective local feature sets of the detection objects from the plurality of local feature graphs, then pooling the local feature sets to obtain respective global behavior features of the detection objects, and reconstructing global information based on multi-view local information to obtain the global behavior features of the detection objects. The information loss caused when the large-size to-be-analyzed image is segmented into the small-size to-be-analyzed sub-images is made up; and finally, obtaining a target object for executing the specific object behavior in the image to be analyzed based on the object behavior text for describing the specific object behavior and the global behavior characteristics of the detection objects.

Description

Background Art

[0002] Multimodal behavior analysis is an important sub-topic of the cross-integration of computer vision and natural language processing. Multimodality refers to the use of multiple types of data or signals (such as images, videos, text, audio, etc.) to process, analyze, and generate information. In recent years, with the rapid development of multimodal technologies represented by Large Language and Vision Assistant (LLAVA), researchers have seen the great potential of open-set human behavior analysis based on multimodal large models.

[0003] Although multimodal technologies represented by LLAVA have developed rapidly and performed well in behavioral analysis, they still face certain challenges in applying them to surveillance and security scenarios. In the actual implementation of these technologies, they still need to overcome two major challenges caused by the fact that human images account for too small a proportion of the overall scene image: first, image analysis takes a long time; second, when simply cutting out independent human images from the overall scene image, the topological relationship and positional relationship between people are easily lost, which in turn brings a series of problems, including impaired information integrity, destruction of spatiotemporal coherence, reduced analysis efficiency, increased risk of misjudgment and missed detection, and affected user experience. These problems not only limit the effectiveness of multimodal technologies in practical applications, but also increase the difficulty of system design and optimization. Summary of the invention

[0004] The embodiments of the present application provide a multimodal behavior analysis method, apparatus, device and storage medium to solve a series of problems caused by the fact that human body images account for too small a proportion of scene images in monitoring security scenes.

[0005] In a first aspect, an embodiment of the present application provides a multimodal behavior analysis method, comprising: Cutting the image to be analyzed to obtain multiple sub-images to be analyzed, wherein one sub-image to be analyzed includes multiple detection objects whose object distance is not greater than a set distance threshold; Perform feature extraction on multiple sub-graphs to be analyzed to obtain multiple local feature graphs; From multiple local feature maps, obtain local feature sets of each detection object respectively, and obtain global behavior features of each detection object by pooling the local feature sets; Based on the object behavior text describing the specific object behavior and the global behavior characteristics of each detected object, the target object performing the specific object behavior in the image to be analyzed is obtained.

[0006] Optionally, the cutting of the image to be analyzed to obtain a plurality of sub-images to be analyzed includes: Performing object detection on the image to be analyzed to obtain an object detection frame for each detected object; Based on each object detection frame, grouping the detection objects to obtain a plurality of object groups; A plurality of detection objects in the same object group in the image to be analyzed are cut into a sub-image to be analyzed, thereby obtaining a plurality of sub-images to be analyzed.

[0007] Optionally, the detection objects are grouped based on the object detection frames to obtain multiple object groups, and the detection objects are traversed in a loop. Each time a detection object is traversed, the following operations are performed: When it is determined that the one detection object is not grouped, expanding the object detection frame of the one detection object outward by a first set step length to obtain a first outwardly expanded detection frame; The one detection object and other detection objects intersecting with the first outward-expanded detection frame are divided into one object group.

[0008] Optionally, the step of cutting a plurality of detection objects in the same object group in the image to be analyzed into a sub-image to be analyzed includes: Expanding the first outward-expanded detection frame of the one detection object outwardly by a second set step length to obtain a second outward-expanded detection frame, wherein the second outward-expanded detection frame covers other detection objects in the same object group as the one detection object; The image to be analyzed is cut along the second outward-expanded detection frame to obtain a sub-image to be analyzed.

[0009] Optionally, the obtaining of respective local feature sets of respective detection objects from the plurality of local feature maps includes: Respectively obtain local behavior features of each detection object in multiple local feature maps; The local behavior features associated with the same object identifier are divided into a local feature set to obtain a local feature set for each detection object.

[0010] Optionally, obtaining the target object that performs the specific object behavior in the image to be analyzed based on the object behavior text describing the specific object behavior and the global behavior features of each detected object includes: Extracting features from the object behavior text describing the behavior of a specific object to obtain text features; Based on the text features and the global behavior features of each detection object, the feature correlation of each detection object is obtained, and the detection object whose feature correlation exceeds the set correlation threshold is determined as the target object that has performed the specific object behavior.

[0011] In a second aspect, the present application also provides a multimodal behavior analysis device, including: An image cutting unit is used to cut the image to be analyzed to obtain multiple sub-images to be analyzed, wherein one sub-image to be analyzed includes multiple detection objects whose object distance is not greater than a set distance threshold; A feature fusion unit is used to extract features from multiple sub-images to be analyzed to obtain multiple local feature images; From multiple local feature maps, obtain local feature sets of each detection object respectively, and obtain global behavior features of each detection object by pooling the local feature sets; The behavior analysis unit is used to obtain the target object performing the specific object behavior in the image to be analyzed based on the object behavior text describing the specific object behavior and the global behavior characteristics of each detected object.

[0012] Optionally, the image cutting unit is used to: Performing object detection on the image to be analyzed to obtain an object detection frame for each detected object; Based on each object detection frame, grouping the detection objects to obtain a plurality of object groups; A plurality of detection objects in the same object group in the image to be analyzed are cut into a sub-image to be analyzed, thereby obtaining a plurality of sub-images to be analyzed.

[0013] Optionally, the image cutting unit loops through the detection objects, and performs the following operations each time a detection object is traversed: When it is determined that the one detection object is not grouped, expanding the object detection frame of the one detection object outward by a first set step length to obtain a first outwardly expanded detection frame; The one detection object and other detection objects intersecting with the first outward-expanded detection frame are divided into one object group.

[0014] Optionally, the image cutting unit is used to: Expanding the first outward-expanded detection frame of the one detection object outwardly by a second set step length to obtain a second outward-expanded detection frame, wherein the second outward-expanded detection frame covers other detection objects in the same object group as the one detection object; The image to be analyzed is cut along the second outward-expanded detection frame to obtain a sub-image to be analyzed.

[0015] Optionally, the feature fusion unit is used to: Respectively obtain local behavior features of each detection object in multiple local feature maps; The local behavior features associated with the same object identifier are divided into a local feature set to obtain a local feature set for each detection object.

[0016] Optionally, the behavior analysis unit is used to: Extracting features from the object behavior text describing the behavior of a specific object to obtain text features; Based on the text features and the global behavior features of each detection object, the feature correlation of each detection object is obtained, and the detection object whose feature correlation exceeds the set correlation threshold is determined as the target object that has performed the specific object behavior.

[0017] In a third aspect, an embodiment of the present application further provides a computer device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned multimodal behavior analysis methods.

[0018] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of any one of the above-mentioned multimodal behavior analysis methods.

[0019] In a fifth aspect, an embodiment of the present application further provides a computer program product, including computer instructions, which are executed by a processor to perform the steps of any of the above-mentioned multimodal behavior analysis methods.

[0020] The beneficial effects of this application are as follows: The embodiments of the present application provide a multimodal behavior analysis method, apparatus, device and storage medium, the method comprising: cutting an image to be analyzed to obtain multiple sub-images to be analyzed, wherein one sub-image to be analyzed includes multiple detection objects whose object distance is not greater than a set distance threshold; performing feature extraction on the multiple sub-images to be analyzed to obtain multiple local feature maps, respectively obtaining local feature sets of each detection object from the multiple local feature maps, and then obtaining global behavior features of each detection object by pooling the local feature sets; finally, obtaining a target object that performs a specific object behavior in the image to be analyzed based on an object behavior text that describes the specific object behavior and the global behavior features of each detection object.

[0021] The embodiment of the present application performs behavior analysis on the basis of the sub-image to be analyzed. The relatively small image scale can not only ensure the detection effect, but also retain the feasibility of performing behavior analysis on complex problems of multi-person interactions in theory. There is no need to worry about whether and how much the small-size image of a single detection object should be expanded, and it also eliminates the interference of large areas where there is no detection object on the behavior analysis effect. Moreover, the embodiment of the present application performs a CLIP forward on the sub-image to be analyzed to extract the local behavior characteristics of multiple people, which can greatly reduce the time consumption and achieve a balance between time consumption and effect.

[0022] The same detection object may exist in multiple sub-images to be analyzed. The feature fusion module reconstructs global information based on multi-view local information by fusing local features from multiple perspectives, thereby compensating for the information loss caused by dividing the large image to be analyzed into small sub-images to be analyzed.

[0023] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of the architecture of a multimodal behavior analysis model provided in an embodiment of the present application; Figure 2 A schematic diagram of the interface of data / label organization and routing settings in the model training phase provided in an embodiment of the present application; Figure 3A A schematic diagram of a process for performing multimodal behavior analysis on a monitoring security scene provided in an embodiment of the present application; Figure 3B A logical schematic diagram of performing multimodal behavior analysis on monitoring security scenarios provided in an embodiment of the present application; Figure 3C A schematic diagram of a process for dividing each detection object into multiple object groups provided in an embodiment of the present application; Figure 3D A logical schematic diagram of grouping a detection object provided in an embodiment of the present application; Figure 3E The embodiment of the present application provides a logic schematic diagram of cutting the image to be analyzed to obtain a sub-image to be analyzed; Figure 3F A logical diagram of the operation of the feature fusion module provided in the embodiment of the present application; Figure 3G A text diagram of an object behavior provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of a multimodal behavior analysis device provided in an embodiment of the present application; Figure 5 A schematic diagram of a hardware structure of a computer device to which an embodiment of the present application is applied; Figure 6A schematic diagram of the hardware composition structure of another computer device to which the embodiment of the present application is applied. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0026] The following is a brief introduction to the design concept of the embodiment of the present application: Multimodal behavior analysis is an important sub-topic of the intersection of computer vision and natural language processing. Multimodality refers to the use of multiple types of data or signals (such as images, videos, text, audio, etc.) to process, analyze, and generate information. In recent years, with the rapid development of multimodal technologies represented by LLAVA, researchers have seen the great potential of open-set human behavior analysis based on large multimodal models.

[0027] Although multimodal technologies represented by LLAVA have developed rapidly and performed well in behavioral analysis, they still face certain challenges in applying them to surveillance and security scenarios. In the actual implementation of these technologies, they still need to overcome two major challenges caused by the fact that human images account for too small a proportion of the overall scene image: first, image analysis takes a long time; second, when simply cutting out independent human images from the overall scene image, the topological relationship and positional relationship between people are easily lost, which in turn brings a series of problems, including impaired information integrity, destruction of spatiotemporal coherence, reduced analysis efficiency, increased risk of misjudgment and missed detection, and affected user experience. These problems not only limit the effectiveness of multimodal technologies in practical applications, but also increase the difficulty of system design and optimization.

[0028] Therefore, in order to solve the above problems, the present application proposes a multimodal behavior analysis method. The method includes: cutting the image to be analyzed to obtain multiple sub-images to be analyzed, and one sub-image to be analyzed includes multiple detection objects whose object distance is not greater than the set distance threshold; extracting features from the multiple sub-images to be analyzed to obtain multiple local feature maps, and obtaining the local feature sets of each detection object from the multiple local feature maps, and then obtaining the global behavior features of each detection object by pooling the local feature sets; finally, based on the object behavior text describing the specific object behavior and the global behavior features of each detection object, the target object performing the specific object behavior in the image to be analyzed is obtained.

[0029] The embodiment of the present application performs behavior analysis on the basis of the sub-image to be analyzed. The relatively small image scale can not only ensure the detection effect, but also retain the feasibility of performing behavior analysis on complex problems of multi-person interactions in theory. There is no need to worry about whether and how much the small-size image of a single detection object should be expanded, and it also eliminates the interference of large areas where there is no detection object on the behavior analysis effect. Moreover, the embodiment of the present application performs a CLIP forward on the sub-image to be analyzed to extract the local behavior characteristics of multiple people, which can greatly reduce the time consumption and achieve a balance between time consumption and effect.

[0030] The same detection object may exist in multiple sub-images to be analyzed. The feature fusion module reconstructs global information based on multi-view local information by fusing local features from multiple perspectives, thereby compensating for the information loss caused by dividing the large image to be analyzed into small sub-images to be analyzed.

[0031] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0032] The present application embodiment designs a multimodal behavior analysis model, such as Figure 1 As shown in the figure, it mainly consists of a target detection sub-model, a greedy scheduling module, a contrastive language-image pre-training (CLIP) visual encoder, a mapper projector, a feature fusion module and a large language model (LLM).

[0033] Among them, the open source target detection sub-model is responsible for object detection in large-size images to be analyzed for monitoring security scenes, and determining the position of each detected object in the image. The greedy scheduling module is responsible for dividing the detected objects with relatively close spatial positions in the large-size images to be analyzed for monitoring security scenes into an object group, providing a cutting benchmark for cutting the image, and obtaining multiple small-size sub-images to be analyzed. The open source CLIP visual encoder is responsible for extracting the visual representation of the sub-image to be analyzed. The projector is a simple linear layer or multi-layer perceptron (Multilayer Perceptron, MLP), which is responsible for completing the alignment of local behavior features to language features. The feature fusion module realizes the reconstruction of global information based on multi-view local information through the fusion of local features of multi-view regions, and compensates for the information loss caused by dividing the large-size image to be analyzed into small-size sub-images to be analyzed. The object behavior text after word segmentation and the visual features aligned by the projector are spliced ​​and combined and sent to the LLM to obtain the corresponding behavior analysis results.

[0034] Different from the model application phase, which uses the images to be analyzed collected in the security monitoring scene as the original form of data, the training samples in the model training phase are small-sized sub-images to be analyzed. The behavior annotation results on the large-sized images to be analyzed are converted into behavior annotation results on the small-sized sub-images to be analyzed as the original form of training samples. At the same time, during the training phase, a conversation may involve multiple sub-images to be analyzed. These sub-images to be analyzed can come from the same image to be analyzed or from different images to be analyzed. Through the arrangement and combination of the sub-images to be analyzed, the data enhancement of the training set is achieved. Behavior analysis is based on people. When the same object appears in multiple sub-images to be analyzed, the multiple sub-images to be analyzed involved in the conversation are analyzed to obtain the local feature set of the object in the image. Then, the local behavior features of the object appearing in multiple sub-images to be analyzed also have multiple sources.

[0035] The open source CLIP visual encoder only performs visual and text alignment at the image level, lacking region-level alignment. If the local behavior features extracted by the native CLIP visual encoder are fused, only suboptimal detection results can be achieved. Therefore, based on the native LLAVA, the embodiment of the present application refines the training process of the multimodal LLM: (1) The first training stage is the same as the native LLAVA, the CLIP visual encoder and LLM are fixed, and only the parameters of the projector are adjusted to roughly achieve alignment from visual features to text features; (2) A new second training stage is added, the parameters of the LLM are fixed, and the parameters of the CLIP visual encoder and projector are adjusted to make CLIP adapt to the data in the security scene field, and at the same time, to achieve the purpose of optimizing its region-level feature extraction capabilities; (3) The third training stage is also the same as the native LLAVA, the CLIP visual encoder and projector are fixed, and only the parameters of the LLM are adjusted.

[0036] For details on data / label organization and routing settings during the model training phase, see Figure 2 , through the logic process set in the routing table, the output of the CLIP visual encoder is connected to the input of the LLM. Any placeholder of local behavior features in the input text of the LLM can be indexed to the corresponding local behavior features in the local feature map associated with the relevant sub-graph to be analyzed. Among them, Figure 2 The value of the first value field represents the question, and the value of the second value field represents the answer. The content not enclosed by angle brackets <> in the question is the participating word segmentation in natural language form. The content enclosed by angle brackets is the placeholder for local behavior features. Before being sent to LLM, they are replaced one by one with the local behavior features of the corresponding detection object in the subgraph to be analyzed. id is the unique identifier of the human body area of ​​the detection object.

[0037] In order to expand the detection range of the training set, the training set is data enhanced, private behavior analysis related data is mixed with the public data set, and special tags are used to distinguish public data from private data. In fact, there are other implementation methods for the logical process of connecting the CLIP visual encoder and the LLM, but they will not be explained in detail. After adopting the above three-stage training, the multimodal behavior analysis model for monitoring security scenarios introduced in the embodiment of this application can be obtained.

[0038] Combination Figure 3A-3B , introduces the process of performing multimodal behavior analysis on monitoring security scenarios based on the method provided in the embodiments of the present application.

[0039] S301: cutting the image to be analyzed to obtain a plurality of sub-images to be analyzed, wherein one sub-image to be analyzed includes a plurality of detection objects whose object distance is not greater than a set distance threshold.

[0040] With the help of the open source target detection sub-model, the image to be analyzed is subjected to object detection, the position of each detection object in the image is located, and the object detection frame of each detection object is obtained. Then, with the help of the greedy scheduling module designed in the embodiment of the present application, each detection object is grouped based on each object detection frame, and each detection object is divided into multiple object groups. Multiple object groups with relatively close spatial positions are grouped into one object group. Therefore, the greedy scheduling module uses the group as the basic unit of cutout, and multiple detection objects in the same object group in the image to be analyzed are cut into a sub-image to be analyzed, and multiple sub-images to be analyzed are obtained.

[0041] Among them, combined Figure 3C-Figure 3D , introducing the process of dividing each detection object into multiple object groups using a cyclic iteration method.

[0042] S3011: Obtain the i-th detection object.

[0043] S3012: Determine whether the i-th detected object is in the grouped set. If so, jump to step 3011; otherwise, execute step 3013.

[0044] S3013: When it is determined that the i-th detection object is not grouped, the object detection frame of the i-th detection object is expanded outward by a first set step length to obtain a first outward expanded detection frame.

[0045] S3014: grouping the i-th detection object and other detection objects intersecting with the first outward expansion detection frame into an object group, and adding the i-th detection object and other detection objects intersecting with the first outward expansion detection frame into a grouped set.

[0046] S3015: Determine whether all detection objects have been traversed, if so, obtain multiple object groups; otherwise, jump to step 3011.

[0047] The embodiment of the present application is to divide the detection object and other detection objects that intersect with the first outward expansion detection frame into an object group, such as Figure 3E As shown, taking the group as the basic unit of cutout, the first outward expansion detection frame of the ith detection object in the group continues to expand outward by a second set step length to obtain a second outward expansion detection frame, and the second outward expansion detection frame covers other detection objects in the same object group as the ith detection object, and then the image to be analyzed is cut along the second outward expansion detection frame to obtain a sub-image to be analyzed.

[0048] Mark the relative positions of multiple detected objects in an object group in the corresponding sub-images to be analyzed. If the same detected object appears in multiple sub-images to be analyzed, the only visual representation of the detected object in the future will be the combination of local behavior features extracted from the above multiple sub-images to be analyzed.

[0049] S302: Extract features from multiple sub-graphs to be analyzed to obtain multiple local feature graphs.

[0050] With the help of the open source CLIP visual encoder, the visual representations of multiple sub-graphs to be analyzed are extracted. With the help of the projector, the local behavior features are aligned to the language features to obtain the local feature maps corresponding to each sub-graph to be analyzed.

[0051] A local feature map contains the local behavior characteristics of each detected object in the image. The local behavior characteristics are also called region of interest features, which refer to the areas selected for further analysis or processing in the image. These areas are considered to contain more useful information than other parts of the image. Only the interesting parts of the image need to be focused on and analyzed, rather than the entire image. This is especially useful when processing large images. By using local behavior features, the image processing time can be significantly reduced, the computational complexity can be reduced, and the processing accuracy can be improved.

[0052] S303: Obtaining local feature sets of each detection object from the multiple local feature maps respectively, and obtaining global behavior features of each detection object by pooling the local feature sets.

[0053] Combination Figure 3F The feature fusion module obtains the local behavior features of each detection object in multiple local feature maps, and divides the local behavior features associated with the same object identifier into a local feature set to obtain the local feature set of each detection object. The feature fusion module performs global average pooling on each local feature set to obtain the global behavior features of each detection object. At this point, each detection object in the large-size image to be analyzed is described by a unique global behavior feature, and the visual preparation work before sending it to the LLM has been completed.

[0054] S304: Based on the object behavior text describing the specific object behavior and the global behavior features of each detected object, a target object performing the specific object behavior in the image to be analyzed is obtained.

[0055] like Figure 3GAs shown, the embodiment of the present application designs an object behavior text for describing the behavior of a specific object. The value of the first value field in the text represents a question, and the value of the second value field represents an answer. The content not enclosed by angle brackets <> in the question is a participating word segmentation in natural language form, and the content enclosed by angle brackets is a placeholder for local features. Before being sent to LLM, they are replaced one by one with the local behavior features of the corresponding detection object in the subgraph to be analyzed. After the object behavior text and the local behavior features are spliced ​​in order, they are sent to LLM for behavior analysis.

[0056] LLM extracts features from the object behavior text that describes the behavior of a specific object to obtain text features. LLM then obtains the feature correlation of each detection object based on the text features and the global behavior features of each detection object, and determines the detection object whose feature correlation exceeds the set correlation threshold as the target object that has performed the specific object behavior.

[0057] like Figure 3G As shown, the LLM output result is also followed by the behavior name, followed by the identification number of the detection object that generated the corresponding behavior. The behavior analysis alarm result of the embodiment of the present application is based on the identification number or the basic alarm unit of a single seat, whether it is a single-person behavior or a multi-person interaction. When there are multiple people in the scene triggering related behaviors, the identification numbers of all people will be output for alarm.

[0058] LLAVA in the related art performs CLIP forward with a large-sized scene original image as the basic unit, or performs CLIP forward with a small-sized image of a single detection object as the basic unit, which cannot guarantee both the processing time and the detection effect. However, the embodiment of the present application performs behavior analysis with the sub-image to be analyzed as the unit. The relatively small image scale can not only guarantee the detection effect, but also retain the feasibility of performing behavior analysis on complex problems of multi-person interactions in theory. There is no need to worry about whether the small-sized image of a single detection object should be expanded and how much to expand, and it also eliminates the interference of large areas where there is no detection object on the behavior analysis effect. Moreover, the embodiment of the present application performs a CLIP forward on the sub-image to be analyzed and extracts the local behavior characteristics of multiple people, which can greatly reduce the time consumption and achieve a balance between time consumption and effect.

[0059] The same detection object may exist in multiple sub-images to be analyzed. The feature fusion module reconstructs global information based on multi-view local information by fusing local features from multiple perspectives, thereby compensating for the information loss caused by dividing the large image to be analyzed into small sub-images to be analyzed.

[0060] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a multimodal behavior analysis device. Figure 4 As shown, the multimodal behavior analysis device 400 may include: An image cutting unit 401 is used to cut the image to be analyzed to obtain multiple sub-images to be analyzed, wherein one sub-image to be analyzed includes multiple detection objects whose object distance is not greater than a set distance threshold; A feature fusion unit 402 is used to extract features from multiple sub-graphs to be analyzed to obtain multiple local feature graphs; From multiple local feature maps, obtain local feature sets of each detection object respectively, and obtain global behavior features of each detection object by pooling the local feature sets; The behavior analysis unit 403 is used to obtain the target object that performs the specific object behavior in the image to be analyzed based on the object behavior text that describes the specific object behavior and the global behavior characteristics of each detection object.

[0061] Optionally, the image cutting unit 401 is used to: Performing object detection on the image to be analyzed to obtain an object detection frame for each detected object; Based on each object detection frame, grouping the detection objects to obtain a plurality of object groups; A plurality of detection objects in the same object group in the image to be analyzed are cut into a sub-image to be analyzed, thereby obtaining a plurality of sub-images to be analyzed.

[0062] Optionally, the image cutting unit 401 loops through the detection objects, and performs the following operations each time a detection object is traversed: When it is determined that the one detection object is not grouped, expanding the object detection frame of the one detection object outward by a first set step length to obtain a first outwardly expanded detection frame; The one detection object and other detection objects intersecting with the first outward-expanded detection frame are divided into one object group.

[0063] Optionally, the image cutting unit 401 is used to: Expanding the first outward-expanded detection frame of the one detection object outwardly by a second set step length to obtain a second outward-expanded detection frame, wherein the second outward-expanded detection frame covers other detection objects in the same object group as the one detection object; The image to be analyzed is cut along the second outward-expanded detection frame to obtain a sub-image to be analyzed.

[0064] Optionally, the feature fusion unit 402 is used to: Respectively obtain local behavior features of each detection object in multiple local feature maps; The local behavior features associated with the same object identifier are divided into a local feature set to obtain a local feature set for each detection object.

[0065] Optionally, the behavior analysis unit 403 is used to: Extracting features from the object behavior text describing the behavior of a specific object to obtain text features; Based on the text features and the global behavior features of each detection object, the feature correlation of each detection object is obtained, and the detection object whose feature correlation exceeds the set correlation threshold is determined as the target object that has performed the specific object behavior.

[0066] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0067] After introducing the multimodal behavior analysis method and apparatus according to an exemplary embodiment of the present application, next, a computer device according to another exemplary embodiment of the present application is introduced.

[0068] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0069] Based on the same inventive concept as the above method embodiment, a computer device is also provided in the embodiment of the present application. In one embodiment, the computer device may be a server. In this embodiment, the structure of the computer device 500 is as follows: Figure 5 As shown, it may include at least a memory 501 , a communication module 503 , and at least one processor 502 .

[0070] The memory 501 is used to store computer programs executed by the processor 502. The memory 501 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0071] The memory 501 may be a volatile memory, such as a random-access memory (RAM); the memory 501 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 501 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 501 may be a combination of the above memories.

[0072] The processor 502 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 502 is configured to implement the multimodal behavior analysis method when calling the computer program stored in the memory 501 .

[0073] The communication module 503 is used to communicate with terminal devices and other servers.

[0074] The specific connection medium between the memory 501, the communication module 503 and the processor 502 is not limited in the embodiment of the present application. Figure 5 In the embodiment, the memory 501 and the processor 502 are connected via a bus 504. The bus 504 is connected to the processor 502 via a bus 504. Figure 5 The connections between the other components are only for illustration and are not intended to be limiting. The bus 504 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 5 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0075] The memory 501 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the parameter calibration method of the gimbal camera of the embodiment of the present application. The processor 502 is used to execute the above-mentioned multimodal behavior analysis method, such as Figure 3A shown.

[0076] In another embodiment, the computer device may also be other computer device terminal devices. In this embodiment, the structure of the computer device may be as follows: Figure 6 As shown, it includes: a communication component 610, a memory 620, a display unit 630, a camera 640, a sensor 650, an audio circuit 660, a Bluetooth module 670, a processor 680 and other components.

[0077] The communication component 610 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the object to send and receive information through the WiFi module.

[0078] The memory 620 can be used to store software programs and data. The processor 680 executes various functions and data processing of the terminal device by running the software programs or data stored in the memory 620. The memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 620 stores an operating system that enables the terminal device to run. In the present application, the memory 620 can store an operating system and various application programs, and can also store a computer program for executing the multimodal behavior analysis method of the embodiment of the present application.

[0079] The display unit 630 can also be used to display information input by the object or information provided to the object and a graphical user interface (GUI) of various menus of the terminal device. Specifically, the display unit 630 may include a display screen 632 arranged on the front of the terminal device. Among them, the display screen 632 can be configured in the form of a liquid crystal display, a light-emitting diode, etc. The display unit 630 can be used to display the code stream access selection interface, the panoramic video display interface, etc. in the embodiment of the present application.

[0080] The display unit 630 can also be used to receive input digital or character information and generate signal input related to the object setting and function control of the terminal device. Specifically, the display unit 630 may include a touch screen 631 arranged on the front of the terminal device, which can collect touch operations of objects on or near it, such as clicking a button, dragging a scroll box, etc.

[0081] The touch screen 631 can be covered on the display screen 632, or the touch screen 631 and the display screen 632 can be integrated to realize the input and output functions of the terminal device, and the integrated display screen can be referred to as a touch display screen. In this application, the display unit 630 can display the application and the corresponding operation steps.

[0082] The camera 60 can be used to capture static images, and the subject can publish the image taken by the camera 640 through the application. The camera 660 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 680 to convert it into a digital image signal.

[0083] The terminal device may further include at least one sensor 650, such as an acceleration sensor 651, a distance sensor 652, a fingerprint sensor 653, and a temperature sensor 654. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0084] The audio circuit 660, the speaker 661, and the microphone 662 can provide an audio interface between the object and the terminal device. The audio circuit 660 can transmit the electrical signal converted from the received audio data to the speaker 661, which is converted into a sound signal for output. The terminal device can also be equipped with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 662 converts the collected sound signal into an electrical signal, which is received by the audio circuit 660 and converted into audio data, and then the audio data is output to the communication component 610 to be sent to, for example, another terminal device, or the audio data is output to the memory 620 for further processing.

[0085] The Bluetooth module 670 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 670 to exchange data.

[0086] The processor 680 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 620 and calling data stored in the memory 620. In some embodiments, the processor 680 may include one or more processing units; the processor 680 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 680. In the present application, the processor 680 can run the operating system, application programs, user interface display and touch response, as well as the multimodal behavior analysis method of the embodiment of the present application. In addition, the processor 680 is coupled to the display unit 630.

[0087] In some possible implementations, various aspects of the multimodal behavior analysis method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the multimodal behavior analysis method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 3A Follow the steps shown in .

[0088] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0089] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0090] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0091] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0092] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user's computer device, partially on the user's computer device, as a separate software package, partially on the user's computer device and partially on a remote computer device, or completely on the remote computer device. In the case of a remote computer device, the remote computer device can be connected to the user's computer device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer device (for example, using an Internet service provider to connect through the Internet).

[0093] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0094] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0095] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0099] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0100] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multimodal behavior analysis method, characterized in that: include: Cutting the image to be analyzed to obtain multiple sub-images to be analyzed, wherein one sub-image to be analyzed includes multiple detection objects whose object distance is not greater than a set distance threshold; Perform feature extraction on multiple sub-graphs to be analyzed to obtain multiple local feature graphs; From multiple local feature maps, obtain local feature sets of each detection object respectively, and obtain global behavior features of each detection object by pooling the local feature sets; Based on the object behavior text describing the specific object behavior and the global behavior characteristics of each detected object, the target object performing the specific object behavior in the image to be analyzed is obtained.

2. The method according to claim 1, characterized in that The cutting of the image to be analyzed to obtain a plurality of sub-images to be analyzed includes: Performing object detection on the image to be analyzed to obtain an object detection frame for each detected object; Grouping the detected objects into a plurality of object groups based on the object detection frames; A plurality of detection objects in the same object group in the image to be analyzed are cut into a sub-image to be analyzed, thereby obtaining a plurality of sub-images to be analyzed.

3. The method according to claim 2, characterized in that The detection objects are grouped based on the object detection frames, and the detection objects are divided into multiple object groups. The detection objects are traversed in a loop, and the following operations are performed each time a detection object is traversed: When it is determined that the one detection object is not grouped, expanding the object detection frame of the one detection object outward by a first set step length to obtain a first outwardly expanded detection frame; The one detection object and other detection objects intersecting with the first outward expanded detection frame are divided into an object group, and the one detection object and other detection objects intersecting with the first outward expanded detection frame are added to the grouped set.

4. The method according to claim 2, characterized in that The step of cutting a plurality of detection objects in the same object group in the image to be analyzed into a sub-image to be analyzed includes: Expanding the first outward-expanded detection frame of the one detection object outwardly by a second set step length to obtain a second outward-expanded detection frame, wherein the second outward-expanded detection frame covers other detection objects in the same object group as the one detection object; The image to be analyzed is cut along the second outward-expanded detection frame to obtain a sub-image to be analyzed.

5. The method according to claim 1, characterized in that The step of obtaining the local feature sets of each detection object from the multiple local feature maps comprises: Respectively obtain local behavior features of each detection object in multiple local feature maps; The local behavior features associated with the same object identifier are divided into a local feature set to obtain a local feature set for each detection object.

6. The method according to claim 1, characterized in that The step of obtaining the target object that performs the specific object behavior in the image to be analyzed based on the object behavior text that describes the specific object behavior and the global behavior features of each detected object includes: Extracting features from the object behavior text describing the behavior of a specific object to obtain text features; Based on the text features and the global behavior features of each detection object, the feature correlation of each detection object is obtained, and the detection object whose feature correlation exceeds the set correlation threshold is determined as the target object that has performed the specific object behavior.

7. A multimodal behavior analysis device, characterized in that: include: An image cutting unit is used to cut the image to be analyzed to obtain multiple sub-images to be analyzed, wherein one sub-image to be analyzed includes multiple detection objects whose object distance is not greater than a set distance threshold; A feature fusion unit is used to extract features from multiple sub-images to be analyzed to obtain multiple local feature images; From multiple local feature maps, obtain local feature sets of each detection object respectively, and obtain global behavior features of each detection object by pooling the local feature sets; The behavior analysis unit is used to obtain the target object performing the specific object behavior in the image to be analyzed based on the object behavior text describing the specific object behavior and the global behavior characteristics of each detected object.

8. A computer device, characterized in that: It includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It includes program code, and when the program code is run on a computer device, the program code is used to enable the computer device to execute the steps of the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images