Taking behavior recognition method and device, electronic equipment and storage medium
By performing human detection and interaction saliency analysis on target images, combined with machine learning models and multimodal technology, the accuracy problem of fighting behavior recognition in complex environments has been solved, achieving fast and accurate fighting behavior recognition and early warning.
Patent Information
- Application Number
- CN202411138090.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2026-03-03
AI Technical Summary
Existing technologies suffer from reduced accuracy in identifying fighting behavior in complex environments such as low light, obstructions, and dense crowds. They struggle to capture subtle behaviors and are susceptible to interference from the external environment.
By performing human detection on the target image, determining the human detection bounding box region, calculating the interaction saliency and identifying the bounding rectangular region, using a machine learning model to identify fighting behavior, and combining a multimodal fighting behavior recognition model and image segmentation technology to eliminate background interference.
It enables rapid and accurate identification of fighting behavior in complex environments, timely issuance of warnings, and prevention of violent incidents.
Smart Images
Figure CN121600587A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, electronic device, and storage medium for recognizing fighting behavior. Background Technology
[0002] In the field of public safety monitoring, the automatic identification of fighting is crucial for preventing violent incidents and maintaining order. Currently, most fight detection relies on video analytics, employing methods such as motion detection and behavioral pattern recognition. While these methods can provide some recognition effectiveness under ideal conditions, they face numerous challenges in practical applications. For example, in complex environments such as low light, obstructions, and dense crowds, the accuracy of fight detection through video analytics significantly decreases. Furthermore, because video surveillance footage contains a large amount of information, it is difficult for video data to capture subtle fighting behaviors, and video data is easily affected by external environmental factors. Summary of the Invention
[0003] This invention provides a method, device, electronic device, and storage medium for identifying fighting behavior, which can quickly and accurately identify fighting behavior.
[0004] According to one aspect of the present invention, a method for identifying fighting behavior is provided, comprising:
[0005] Perform human detection on the target image and determine the human detection bounding box region for each human body contained in the target image;
[0006] When the number of human detection box regions is greater than 1, each human detection box region in the target image is traversed, and the interaction saliency between the first human in the current human detection box region and the second human in each target human detection box region is determined; wherein, the target human detection box region is other human detection box regions in the target image other than the current human detection box region.
[0007] When the interaction salience is greater than a preset salience threshold, the bounding rectangle region of the first human body and the second human body is determined in the target image, and the bounding rectangle region is used as the target human body interaction image;
[0008] Based on the target human interaction image, it is determined whether the first human and the second human are engaged in fighting.
[0009] According to another aspect of the present invention, a fighting behavior recognition device is provided, comprising:
[0010] The human body detection bounding box region determination module is used to perform human body detection on the target image and determine the human body detection bounding box region for each human body contained in the target image.
[0011] An interaction saliency determination module is used to, when the number of human detection box regions is greater than 1, traverse each human detection box region in the target image and determine the interaction saliency between the first human in the current human detection box region and the second human in each target human detection box region; wherein, the target human detection box region is other human detection box regions in the target image other than the current human detection box region.
[0012] The target human interaction image determination module is used to determine the bounding rectangular region of the first human body and the second human body in the target image when the interaction salience is greater than a preset salience threshold, and to use the bounding rectangular region as the target human interaction image.
[0013] The fighting behavior recognition module is used to identify whether the first human body and the second human body are fighting based on the target human body interaction image.
[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0015] At least one processor; and
[0016] A memory communicatively connected to the at least one processor; wherein,
[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the fighting behavior recognition method according to any embodiment of the present invention.
[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the fighting behavior recognition method according to any embodiment of the present invention.
[0019] The fighting behavior recognition scheme of this invention includes: performing human body detection on a target image to determine the human body detection box region for each human body contained in the target image; when the number of human body detection box regions is greater than 1, traversing each human body detection box region in the target image, and determining the interaction saliency between a first human body in the current human body detection box region and a second human body in each target human body detection box region; wherein, the target human body detection box region is any other human body detection box region in the target image other than the current human body detection box region; when the interaction saliency is greater than a preset saliency threshold, determining the bounding rectangle region of the first human body and the second human body in the target image, and using the bounding rectangle region as the target human body interaction image; and identifying whether the first human body and the second human body are fighting based on the target human body interaction image. The technical solution provided by this invention can quickly and accurately identify fighting behavior, thereby facilitating timely warnings and preventing violent incidents.
[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A flowchart of a method for recognizing fighting behavior provided in an embodiment of the present invention;
[0023] Figure 2 A flowchart of a method for recognizing fighting behavior provided in another embodiment of the present invention;
[0024] Figure 3 A flowchart of a method for recognizing fighting behavior provided in another embodiment of the present invention;
[0025] Figure 4 This is a schematic diagram of the structure of a fighting behavior recognition device provided in an embodiment of the present invention;
[0026] Figure 5 A schematic diagram of the structure of an electronic device for implementing the fighting behavior recognition method of this invention. Detailed Implementation
[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0029] Figure 1 This is a flowchart illustrating a method for recognizing fighting behavior according to an embodiment of the present invention. This embodiment is applicable to situations requiring the recognition of fighting behavior. The method can be executed by a fighting behavior recognition device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:
[0030] S110. Perform human detection on the target image and determine the human detection box region for each human body contained in the target image.
[0031] In this embodiment of the invention, when a fighting behavior recognition command is detected, a target image is acquired, wherein the target image is a detection image used to identify whether a fighting behavior exists. Human detection is performed on the target image based on a human detection algorithm to determine the human detection box for each human body contained in the target image, and the region corresponding to each human detection box in the target image is taken as the human detection box region. It can be understood that one human body corresponds to one human detection box region, and the number of human bodies contained in the target image is the same as the number of human detection box regions. For example, human detection is performed on the target image based on the GroundingDino model to obtain the human detection box region for each human body contained in the target image. GroundingDino is an open-set object detection model that enables human detection. The GroundingDino model allows for rapid human detection of the target image, thereby determining the human detection box region for each human body contained in the target image.
[0032] S120. When the number of human detection box regions is greater than 1, traverse each human detection box region in the target image and determine the interaction saliency between the first human in the current human detection box region and the second human in each target human detection box region; wherein, the target human detection box region is other human detection box regions in the target image other than the current human detection box region.
[0033] In this embodiment of the invention, the number of human detection box regions is determined. If the number of human detection box regions is less than 1 (i.e., 0 or 1), it is determined that the number of human bodies in the target image is insufficient to trigger a fighting conflict event. If the number of human detection box regions is greater than 1, that is, at least two human detection box regions are detected, then each human detection box region needs to be further analyzed to determine whether there is a possible fighting behavior.
[0034] For example, when the number of human detection bounding box regions is greater than 1, each human detection bounding box region in the target image is traversed, and other human detection bounding box regions in the target image other than the current human detection bounding box region are taken as target human detection bounding box regions. The interaction saliency between the first human in the current human detection bounding box region and the second human in each target human detection bounding box region is then determined. The interaction saliency reflects the probability of interaction between the first human in the current human detection bounding box region and the second human in the target human detection bounding box region. For example, there are three human detection bounding box regions: human detection bounding box region A, human detection bounding box region B, and human detection bounding box region C. If human detection bounding box region A is the current human detection bounding box region, then the interaction saliency between human a in human detection bounding box region A and human b in human detection bounding box region B, and the interaction saliency between human a in human detection bounding box region A and human c in human detection bounding box region C are determined. Similarly, taking human detection box region B as the current human detection box region, the interaction significance between human body b in human detection box region B and human body a in human detection box region A is determined, as is the interaction significance between human body b in human detection box region B and human body c in human detection box region C. Taking human detection box region C as the current human detection box region, the interaction significance between human body c in human detection box region C and human body a in human detection box region A is determined, as is the interaction significance between human body c in human detection box region C and human body b in human detection box region B.
[0035] Optionally, the area of the intersection region between the current human detection box region and the target human detection box region can be calculated, and the area of the intersection region can be used as the interaction salience between the first human in the current human detection box region and the second human in the target human detection box region. Optionally, the ratio of the area of the intersection region between the current human detection box region and the target human detection box region to the area of the current human detection box region can also be used as the interaction salience between the first human in the current human detection box region and the second human in the target human detection box region.
[0036] S130. When the interaction salience is greater than a preset salience threshold, the bounding rectangular region of the first human body and the second human body is determined in the target image, and the bounding rectangular region is used as the target human body interaction image.
[0037] In this embodiment of the invention, it is determined whether the interaction saliency between the first human body in the current human body detection box region and the second human body in each target human body detection box region is greater than a preset saliency threshold. If so, it indicates that the probability of a conflict event between the first human body in the current human body detection box region and the second human body in the target human body detection box region is relatively high. Therefore, when the interaction saliency is greater than the preset saliency threshold, the bounding rectangle (BoundingBox) of the first human body and the second human body is determined. The region corresponding to the bounding rectangle (BoundingBox) in the target image is taken as the bounding rectangle region of the first human body and the second human body, and the bounding rectangle region is taken as the target human body interaction image.
[0038] S140. Based on the target human interaction image, identify whether the first human and the second human are fighting.
[0039] In this embodiment of the invention, a fighting behavior recognition algorithm is used to analyze a target human interaction image to identify whether a first human and a second human in the target human interaction image are engaging in fighting behavior. For example, the target human interaction image is input into a pre-built fighting behavior recognition model, which is a machine learning model capable of quickly and accurately identifying whether two human figures in an image are fighting. Based on the output of the fighting behavior recognition model, it is determined whether a first human and a second human in the target human interaction image are engaging in fighting behavior.
[0040] Optionally, identifying whether the first and second human beings are fighting based on the target human interaction image includes: obtaining a human mask image corresponding to the target human interaction image; performing a pixel-by-pixel logical AND operation on the human mask image and the target human interaction image to generate a human foreground image; and identifying whether the first and second human beings are fighting based on the human foreground image. For example, obtaining the human mask image corresponding to the target human interaction image defines the precise contour of the human body region in the target human interaction image, where the human body region is marked as 1 and the non-human body region is marked as 0. For example, inputting the bounding rectangles (BoundingBoxes) of the first and second human beings and the target image into a SAM model, and obtaining the human mask image through the SAM model. SAM (Segment Anything Model) is an image segmentation model capable of segmenting any object in any image without any annotation. By performing a pixel-by-pixel logical AND operation between the human body mask image and the target human body interaction image, the human body region can be effectively extracted from the target human body interaction image, generating a human body foreground image. This ensures that the target human body interaction image contains only the human body portion, removing all background interference. The human body foreground image is then input into a fighting behavior recognition model. Based on the model's output, it is determined whether the first and second human bodies in the foreground image are engaged in a fight. This setup effectively eliminates background interference in fighting behavior recognition, further improving the accuracy of the recognition.
[0041] The method for identifying fighting behavior according to embodiments of the present invention includes: performing human body detection on a target image to determine the human body detection box region for each human body contained in the target image; when the number of human body detection box regions is greater than 1, traversing each human body detection box region in the target image, and determining the interaction saliency between a first human body in the current human body detection box region and a second human body in each target human body detection box region; wherein, the target human body detection box region is any other human body detection box region in the target image other than the current human body detection box region; when the interaction saliency is greater than a preset saliency threshold, determining the bounding rectangle region of the first human body and the second human body in the target image, and using the bounding rectangle region as the target human body interaction image; and identifying whether the first human body and the second human body are fighting based on the target human body interaction image. The technical solution provided by the embodiments of the present invention can quickly and accurately identify fighting behavior, thereby facilitating timely warnings and preventing violent incidents.
[0042] Figure 2 A flowchart of a fighting behavior recognition method provided in another embodiment of the present invention is shown below. Figure 2As shown, the method includes:
[0043] S210. Perform human detection on the target image and determine the human detection box region for each human body contained in the target image.
[0044] S220. When the number of human detection box regions is greater than 1, traverse each human detection box region in the target image, determine a search region centered on the current human detection box region, and divide the search region into N grid regions of the same size as the current human detection box region; where N is an integer greater than or equal to 2.
[0045] In this embodiment of the invention, when the number of human detection box regions is greater than 1, each human detection box region in the target image is traversed to determine a search region centered on the current human detection box region. It can be understood that the search region is the area centered on the center of the current human detection box region, and the search region is a larger area containing the current human detection box region. For example, the width of the search region is twice the width of the current human detection box region, and the height of the search region is twice the height of the current human detection box region. The search region is divided into N equal grid regions of the same size as the current human detection box region; where N is an integer greater than or equal to 2. It should be noted that the N grid regions may or may not overlap. When the N grid regions overlap, adjacent grid regions may overlap in the width direction or in the height direction.
[0046] Optionally, the search area is a rectangular area with a width m times the width of the current human detection box area and a length n times the length of the current human detection box area. Any two vertically adjacent grid areas among the N grid areas overlap; where m is greater than or equal to n, and both m and n are integers greater than or equal to 1. For example, the width of the search area is 3 times the width of the current human detection box area, and the height of the search area is twice the height of the current human detection box area. The search area is divided into 9 equal grid areas according to the size of the current human detection box area, where the width and height of each grid area are the same as the width and height of the current human detection box area. Since each grid area in the search area is divided according to the size of the previous human detection box area, and the height of the search area is twice the height of the current human detection box area, any two vertically adjacent grid areas among the 9 grid areas will overlap by H / 2, where H is the height of the current human detection box area. The advantage of this setting is that it effectively ensures that the entire search area is covered without omissions.
[0047] S230. Take the other human detection box regions in the target image other than the current human detection box region as the target human detection box regions, and calculate the initial region intersection ratio between the target human detection box region and each of the grid regions for each target detection box region.
[0048] In this embodiment of the invention, all human detection box regions other than the current human detection box region are designated as target human detection box regions. For each target detection box region, an initial region intersection ratio between the target human detection box region and each grid region is calculated. The initial region intersection ratio reflects the size of the intersection area between the target human detection box region and each grid region. Optionally, calculating the initial region intersection ratio between the target human detection box region and each grid region includes: determining the area of the intersection area between the target human detection box region and each grid region, and determining the target area of the target human detection box region; calculating the initial region intersection ratio between the target human detection box region and each grid region based on the intersection area and the target area; wherein the initial region intersection ratio is the ratio of the intersection area to the target area.
[0049] For example, for each of the N grid regions, the area of the intersection region between the target human detection box region and the grid region is determined. The target area of the target human detection box region is determined, and the ratio of the intersection region area to the target area is used as the initial region intersection ratio between the target human detection box region and the grid regions. For example, the current human detection box region is human detection box region A, and the N grid regions corresponding to human detection box region A are A1, A2, A3, A4, A5, A6, A7, A8, A9, A1, A2, A3, A4, A5, A6, A7, A8, A9, A1, A2, A1, A2, A3, A4, A1, A2, A3, A4, A1, A2, A3, A4, A1, A2, A3, A4, A1, A2, A3, A4, A1, A2, A3, A4, A4, A5, A6, A7, A8, A9, A1, A1, A2 ...4, A1, A4, A5, A6, A7, A8, A9, A1, A4, A8, A9, A1, A4, A8, 11 A 12 A 21 A 22 A ij ..., where the target human detection bounding box region is human detection bounding box region B, the initial region intersection ratio of each grid region corresponding to human detection bounding box region B and human detection bounding box region A can be calculated using the following formula: Intersection Ratio (IR) ij This represents the initial region intersection ratio of the grid region Aij in the i-th row and j-th column of the search region corresponding to the human detection box region B and the human detection box region A. AreaBoxB represents the target area of the human detection box region B, and Area(BoxB∩Aij) represents the area of the intersection region of the grid region Aij in the i-th row and j-th column of the search region corresponding to the human detection box region B and the human detection box region A.
[0050] S240. Calculate the interaction saliency between the first human body in the current human body detection box region and the second human body in the target human body detection box region based on the initial region intersection ratio.
[0051] In this embodiment of the invention, the average of the initial region intersection ratios between the target human detection box region and all grid regions can be used as the interaction saliency between the first human in the current human detection box region and the second human in the target human detection box region. Optionally, calculating the interaction saliency between the first human in the current human detection box region and the second human in the target human detection box region based on the initial region intersection ratio includes: determining the region weight of each grid region; wherein the sum of the region weights of N grid regions is 1; for each grid region, calculating the product of the initial region intersection ratio and the region weight corresponding to the grid region, and using the product as the target region intersection ratio; and using the sum of the target region intersection ratios corresponding to the N grid regions as the interaction saliency between the first human in the current human detection box region and the second human in the target human detection box region. The advantage of this setting is that the spatial relationship between the current human detection box region and the target human detection box region can be accurately evaluated, so as to more accurately reflect the interaction saliency between the first human in the current human detection box region and the second human in the target human detection box region, thereby truly reflecting the probability of interaction between the first human and the second human through the interaction saliency.
[0052] In this embodiment of the invention, the region weight of each of the N grid regions is determined, where the region weights of the N grid regions can be represented in the form of a weight matrix. For example, dividing the search area into 9 equal grid regions according to the size of the current human detection box region allows for the determination of a 3×3 weight matrix W. Each element of the weight matrix W corresponds to the region weight of each grid region in the search area, representing not only the importance of each grid region when calculating the saliency of the interaction between the first and second human bodies, but also the probability of human interaction behavior occurring in each grid region. For the four corner elements w of the weight matrix W... 11 w 31 w 13 w 33 The assigned weight for the region can be 0.1 because the target human detection box region and the current human detection box region may intersect in planar positions within the four grid regions, and the first human in the current human detection box region and the second human in the target human detection box region may have limb contact in spatial positions within these four grid regions. The second element w in the first and third columns of the weight matrix... 21 and w 32Each region is assigned a weight of 0.3. This is because the target human detection box region and the current human detection box region have a high probability of intersection in both planar and spatial positions within these two grid regions. Simultaneously, these two grid regions are also potential hotspots for interaction between the first human in the current human detection box region and the second human in the target human detection box region. This is achieved by assigning a weight of 0.3 to the second element w in the first and third columns of the weight matrix. 21 and w 32 Assigning a weight of 0.3 to these two grid regions highlights their importance in the fight detection process and their priority relative to other grid regions. The remaining elements of the weight matrix are set to 0 because the probability of the target human detection box region intersecting with the current human detection box region in the planar and spatial positions of other grid regions is very small. Therefore, the weight matrix W can be represented as... By setting the weight matrix as described above, the spatial relationship between the current human detection box region and the target human detection box region can be accurately evaluated, as well as the potential interaction between the first human body in the current human detection box region and the second human body in the target human detection box region.
[0053] In this embodiment of the invention, the product of the initial region intersection ratio corresponding to the grid region and the region weights corresponding to the grid region in the weight matrix is calculated, and the product is used as the target region intersection ratio. For example, the target region intersection ratio is calculated according to the following formula: Weighted IR ij =IR ij ×W ij Among them, Weighted IR ij IR represents the intersection ratio of the target region with the grid region in the i-th row and j-th column of the search region corresponding to the current human detection bounding box region. ij W represents the initial region intersection ratio of the grid region in the i-th row and j-th column of the search region corresponding to the current human detection box region. ij This represents the element in the i-th row and j-th column of the weight matrix W. Calculate the sum of the intersection ratios of the target regions corresponding to the N grid regions, and use this sum as the interaction saliency between the first human in the current human detection box region and the second human in the target human detection box region.
[0054] S250. When the interaction salience is greater than a preset salience threshold, the bounding rectangular region of the first human body and the second human body is determined in the target image, and the bounding rectangular region is used as the target human body interaction image.
[0055] S260. Based on the target human interaction image, identify whether the first human and the second human are fighting.
[0056] The fighting behavior recognition method of this invention can accurately calculate the interaction salience between the first human in the current human detection box area and the second human in the target human detection box area based on the initial region intersection ratio between each grid region in the search area corresponding to the current human detection box area and the target human detection box area, thereby further improving the accuracy of fighting behavior recognition.
[0057] Figure 3 A flowchart of a method for recognizing fighting behavior provided in another embodiment of the present invention is shown below. Figure 3 As shown, the method includes:
[0058] S310. Perform human detection on the target image and determine the human detection box region for each human body contained in the target image.
[0059] S320. When the number of human detection box regions is greater than 1, traverse each human detection box region in the target image and determine the interaction saliency between the first human in the current human detection box region and the second human in each target human detection box region; wherein, the target human detection box region is other human detection box regions in the target image other than the current human detection box region.
[0060] S330. When the interaction salience is greater than a preset salience threshold, the bounding rectangular region of the first human body and the second human body is determined in the target image, and the bounding rectangular region is used as the target human body interaction image.
[0061] S340. Determine at least two target text descriptions corresponding to the target human body interaction image.
[0062] In this embodiment of the invention, at least two target text descriptions corresponding to the target human interaction image are determined. These target text descriptions are used to subsequently indicate whether there is a fight between the two people in the target human interaction image. For example, the target text descriptions may include: "A picture of people pushing each other," "People arguing in the picture," and "Two people talking in the picture." Optionally, the at least two target text descriptions corresponding to the target human interaction image may include at least one positive target text description and at least one negative target text description. The positive target text description contains positive sample verb phrases positively correlated with fighting behavior, such as "A picture of people pushing each other." The negative target text description contains negative sample verb phrases negatively correlated with fighting behavior, such as "Two people talking in the picture." Optionally, the positive sample verb phrases in the positive target text description may be empty strings, or the negative sample verb phrases in the negative target text description may also be empty strings.
[0063] S350. Input the target human interaction image and the at least two target text descriptions into a pre-constructed multimodal fighting behavior recognition model, and determine whether the first human and the second human are fighting based on the output of the multimodal fighting behavior recognition model; wherein, the multimodal fighting behavior recognition model is a fighting behavior recognition model generated by training a multimodal model based on an image-text pair training set, and the image-text pair training set contains at least two fighting sample images and key feature text descriptions corresponding to each fighting sample image.
[0064] In this embodiment of the invention, a pre-constructed multimodal fighting behavior recognition model is obtained. This model can be a multimodal learning model (Contrastive Language-Image Pre-training, CLIP), where the CLIP model can understand the relationship between image content and corresponding text descriptions. The construction process of the multimodal fighting behavior recognition model may include: S1, acquiring at least two fighting sample images, including positive and negative fighting sample images. S2, determining a key feature text description corresponding to each fighting sample image, and using the key feature text description as a label for the corresponding fighting sample image, where the key feature text description reflects the key features in the fighting sample image. S3, pairing each fighting sample image with its corresponding key feature text description to form image-text pairs, and generating an image-text pair training set based on at least two image-text pairs. S4. The CLIP model is trained using an image-text pair training set to generate a multimodal fighting behavior recognition model. The image-text pair training set guides the CLIP model in learning the correlation between fighting sample images and key feature text descriptions. The Chinese CLIP-L-14@336 model is selected as the foundation. The CLIP model has already been trained on a large-scale dataset and possesses good feature extraction capabilities. Based on this, the learning rate and optimization algorithm of the CLIP model are adjusted using the image-text pair training set to fine-tune the CLIP model and generate the multimodal fighting behavior recognition model.
[0065] The target human interaction image and at least two target text descriptions are input into a pre-built multimodal fighting behavior recognition model. The multimodal fighting behavior recognition model maps the target human interaction image and at least two target text descriptions to a common feature space, generating a feature vector z = fCLIP(I, P; Θ), where I represents the input target human interaction image, P represents the at least two target text descriptions corresponding to the target human interaction image, fCLIP represents the fine-tuned CLIP model (i.e., the multimodal fighting behavior recognition model), and Θ represents the model parameters of the multimodal fighting behavior recognition model. The feature vector is normalized using the softmax function g() to obtain the probability distribution for each category. Among them, z c This represents the element in the feature vector corresponding to the c-th category, where k represents the index of the category. The category c with the highest probability is selected. max As the classification result of the multimodal fighting behavior recognition model: c max =argmax c (π c This process can be represented by the following unified formula:
[0066]
[0067] The fighting behavior recognition method of this invention uses the target human interaction image and at least two target text descriptions as input to a multimodal fighting behavior recognition model. The multimodal fighting behavior recognition model can quickly and accurately identify whether there is a fighting behavior between the first human in the current human detection box area and the second human in the target human detection box area.
[0068] In some embodiments, determining at least two target text descriptions corresponding to the target human interaction image includes: extracting verb phrases from the key feature text descriptions corresponding to each fighting sample image in the image-text pair training set; determining the frequency distribution of the verb phrases, and selecting at least two target verbs with the highest frequency according to the frequency distribution; embedding the at least two target verbs into a pre-set standard text prompt template to generate at least two target text descriptions corresponding to the target human interaction image.
[0069] For example, dependency parsing is performed on the key feature text descriptions corresponding to each fighting sample image to extract verb phrases from each key feature text description. For instance, the Stanza toolkit can be used for dependency parsing of the key feature text descriptions. This involves first performing word segmentation on the key feature text descriptions, and then performing part-of-speech tagging on the segmentation results to identify verb phrases. For phrases tagged as verbs, their dependency roles in the dependency tree are analyzed. If the verb's dependency role is one of the following: sentence root, part of a parallel structure (conj), clause complement (ccomp), or parataxis, then the object component that depends on the verb is further tracked and identified.
[0070] Determine the frequency distribution of all verb phrases involved in the image-text pair training set. Based on the frequency distribution, select at least two target verbs with the highest frequency, such as selecting the five most frequent verb phrases as target verbs. Embed at least two target verbs into predefined standard text prompt templates, so that each target verb is combined with the standard text prompt template to generate a target text description. For example, the standard text prompt template can be: "There is someone [action] in the picture," or "A picture of someone [action]." Table 1 shows the frequency distribution of all verb phrases involved in the image-text pair training set provided by this embodiment of the invention.
[0071] Table 1 shows the frequency distribution of all verb phrases involved in the training set.
[0072]
[0073] Based on the frequency distribution of verb phrases shown in Table 1, the five most frequent target verbs were selected: push, fight, walk, look at a phone, and talk. These five target verbs were then embedded into a standard text prompt template to generate five target text descriptions. For example, if the standard text prompt template is “There is someone [action] in the picture,” the five generated target text descriptions would be: “There is someone [pushing] in the picture,” “There is someone [fighting] in the picture,” “There is someone [walking] in the picture,” “There is someone [looking at a phone] in the picture,” and “There is someone [talking] in the picture.” Optionally, the frequency distribution of all verb phrases in the training set may be biased in the determined image and text. Therefore, the frequency distribution of verb phrases can be manually corrected. As shown in Table 1, the verb phrase “take a phone” is deleted.
[0074] The fighting behavior recognition method of this invention performs dependency parsing on the text label set in the training set, extracts specific parts of speech and specific structures through a syntactic tree, and labels the required verb phrases. Detailed frequency statistical analysis of the extracted verb phrases can identify and quantify common human movement patterns in reality. A customized strategy is used to determine targeted semantic prompts as text input, guiding and optimizing the model to accurately capture the semantic relationship between images and text, thereby improving the accuracy of fighting behavior recognition.
[0075] Figure 4 This is a schematic diagram of a fighting behavior recognition device provided in an embodiment of the present invention. Figure 4 As shown, the device includes:
[0076] The human body detection box region determination module 410 is used to perform human body detection on the target image and determine the human body detection box region of each human body contained in the target image.
[0077] The interaction saliency determination module 420 is used to traverse each of the human body detection box regions in the target image when the number of human body detection box regions is greater than 1, and determine the interaction saliency between the first human body in the current human body detection box region and the second human body in each target human body detection box region; wherein, the target human body detection box region is other human body detection box regions in the target image other than the current human body detection box region.
[0078] The target human interaction image determination module 430 is used to determine the bounding rectangular region of the first human body and the second human body in the target image when the interaction salience is greater than a preset salience threshold, and to use the bounding rectangular region as the target human interaction image.
[0079] The fighting behavior recognition module 440 is used to identify whether the first human body and the second human body are fighting based on the target human body interaction image.
[0080] Optionally, the interaction saliency determination module includes:
[0081] A grid region division unit is used to traverse each human detection box region in the target image, determine a search region centered on the current human detection box region, and divide the search region into N grid regions of the same size as the current human detection box region; where N is an integer greater than or equal to 2.
[0082] The initial region intersection ratio calculation unit is used to calculate the initial region intersection ratio between the target human detection box region and each of the grid regions for each of the target detection box regions;
[0083] The interaction saliency calculation module is used to calculate the interaction saliency between the first human body in the current human body detection box region and the second human body in the target human body detection box region based on the initial region intersection ratio.
[0084] Optionally, the initial region intersection ratio calculation unit is used for:
[0085] Determine the area of the intersection region between the target human detection frame region and each of the grid regions, and determine the target area of the target human detection frame region;
[0086] Based on the area of the intersection region and the target area, calculate the initial region intersection ratio between the target human detection frame region and each of the grid regions; wherein, the initial region intersection ratio is the ratio of the area of the intersection region to the target area.
[0087] Optionally, the interaction saliency calculation module is used for:
[0088] Determine the region weight for each of the grid regions; wherein the sum of the region weights of the N grid regions is 1;
[0089] For each of the grid regions, the product of the initial region intersection ratio and the region weight is calculated, and the product is used as the target region intersection ratio.
[0090] The sum of the intersection ratios of the target regions corresponding to the N grid regions is used as the interaction saliency between the first human body in the current human body detection box region and the second human body in the target human body detection box region.
[0091] Optionally, the search area is a rectangular area with a width that is m times the width of the current human detection box area and a length that is n times the length of the current human detection box area. Any two adjacent grid areas in the vertical direction among the N grid areas have overlapping areas; where m is greater than or equal to n, and both m and n are integers greater than or equal to 1.
[0092] Optionally, the fighting behavior recognition module is used for:
[0093] Obtain the human mask image corresponding to the target human body interaction image;
[0094] Perform a pixel-by-pixel logical AND operation between the human body mask image and the target human body interaction image to generate a human body foreground image;
[0095] Based on the foreground image of the human body, it is determined whether the first human body and the second human body are fighting.
[0096] Optionally, the fighting behavior recognition module includes:
[0097] A target text description determination unit is used to determine at least two target text descriptions corresponding to the target human interaction image;
[0098] A fighting behavior recognition unit is used to input the target human interaction image and the at least two target text descriptions into a pre-constructed multimodal fighting behavior recognition model, and determine whether the first human and the second human are fighting based on the output of the multimodal fighting behavior recognition model; wherein, the multimodal fighting behavior recognition model is a fighting behavior recognition model generated by training a multimodal model based on an image-text pair training set, and the image-text pair training set contains at least two fighting sample images and key feature text descriptions corresponding to each fighting sample image.
[0099] Optionally, the target text description determining unit is used for:
[0100] Extract verb phrases from the key feature text descriptions corresponding to each fighting sample image in the training set;
[0101] Determine the frequency distribution of the verb phrases, and select at least two target verbs with the highest frequency based on the frequency distribution;
[0102] The at least two target verbs are embedded into a pre-defined standard text prompt template to generate at least two target text descriptions corresponding to the target human body interaction image.
[0103] The fighting behavior recognition device provided in this embodiment of the invention can execute the fighting behavior recognition method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.
[0104] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0105] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0106] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0107] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the fighting behavior recognition method.
[0108] In some embodiments, the fighting behavior recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the fighting behavior recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the fighting behavior recognition method by any other suitable means (e.g., by means of firmware).
[0109] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0110] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0111] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0114] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0115] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for recognizing fighting behavior, characterized in that, include: Perform human detection on the target image and determine the human detection bounding box region for each human body contained in the target image; When the number of human detection box regions is greater than 1, each human detection box region in the target image is traversed, and the interaction saliency between the first human in the current human detection box region and the second human in each target human detection box region is determined; wherein, the target human detection box region is other human detection box regions in the target image other than the current human detection box region. When the interaction salience is greater than a preset salience threshold, the bounding rectangle region of the first human body and the second human body is determined in the target image, and the bounding rectangle region is used as the target human body interaction image; Based on the target human interaction image, it is determined whether the first human and the second human are engaged in fighting.
2. The method according to claim 1, characterized in that, Traverse each human detection box region in the target image, and determine the interaction saliency between the first human in the current human detection box region and the second human in each target human detection box region, including: Traverse each human detection box region in the target image, determine a search region centered on the current human detection box region, and divide the search region into N grid regions of the same size as the current human detection box region; where N is an integer greater than or equal to 2; For each of the target detection box regions, calculate the initial region intersection ratio between the target human body detection box region and each of the grid regions; The interaction saliency between the first human body in the current human body detection box region and the second human body in the target human body detection box region is calculated based on the initial region intersection ratio.
3. The method according to claim 2, characterized in that, The interaction saliency between the first human body in the current human body detection box region and the second human body in the target human body detection box region is calculated based on the initial region intersection ratio, including: Determine the region weight for each of the grid regions; wherein the sum of the region weights of the N grid regions is 1; For each of the grid regions, the product of the initial region intersection ratio and the region weight is calculated, and the product is used as the target region intersection ratio. The sum of the intersection ratios of the target regions corresponding to the N grid regions is used as the interaction saliency between the first human body in the current human body detection box region and the second human body in the target human body detection box region.
4. The method according to claim 2 or 3, characterized in that, The search area is a rectangular area with a width that is m times the width of the current human detection box area and a length that is n times the length of the current human detection box area. Any two adjacent grid areas in the vertical direction among the N grid areas have overlapping areas; where m is greater than or equal to n, and both m and n are integers greater than or equal to 1.
5. The method according to claim 1, characterized in that, Identifying whether the first human and the second human are fighting based on the target human interaction image includes: Obtain the human mask image corresponding to the target human body interaction image; Perform a pixel-by-pixel logical AND operation between the human body mask image and the target human body interaction image to generate a human body foreground image; Based on the foreground image of the human body, it is determined whether the first human body and the second human body are fighting.
6. The method according to claim 1, characterized in that, Identifying whether the first human and the second human are fighting based on the target human interaction image includes: Determine at least two target text descriptions corresponding to the target human interaction image; The target human interaction image and the at least two target text descriptions are input into a pre-constructed multimodal fighting behavior recognition model. Based on the output of the multimodal fighting behavior recognition model, it is determined whether the first human and the second human are fighting. The multimodal fighting behavior recognition model is a fighting behavior recognition model generated by training a multimodal model based on an image-text pair training set. The image-text pair training set contains at least two fighting sample images and key feature text descriptions corresponding to each fighting sample image.
7. The method according to claim 6, characterized in that, Determine at least two target text descriptions corresponding to the target human interaction image, including: Extract verb phrases from the key feature text descriptions corresponding to each fighting sample image in the training set; Determine the frequency distribution of the verb phrases, and select at least two target verbs with the highest frequency based on the frequency distribution; The at least two target verbs are embedded into a pre-defined standard text prompt template to generate at least two target text descriptions corresponding to the target human body interaction image.
8. A fighting behavior recognition device, characterized in that, include: The human body detection bounding box region determination module is used to perform human body detection on the target image and determine the human body detection bounding box region for each human body contained in the target image. An interaction saliency determination module is used to, when the number of human detection box regions is greater than 1, traverse each human detection box region in the target image and determine the interaction saliency between the first human in the current human detection box region and the second human in each target human detection box region; wherein, the target human detection box region is other human detection box regions in the target image other than the current human detection box region. The target human interaction image determination module is used to determine the bounding rectangular region of the first human body and the second human body in the target image when the interaction salience is greater than a preset salience threshold, and to use the bounding rectangular region as the target human interaction image. The fighting behavior recognition module is used to identify whether the first human body and the second human body are fighting based on the target human body interaction image.
9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the fighting behavior recognition method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the fighting behavior recognition method according to any one of claims 1-7.