Monitoring image processing method and apparatus, and non-volatile computer storage medium

Abstract

The present application belongs to the technical field of monitoring. Disclosed are a monitoring image processing method and apparatus, and a non-volatile computer storage medium. The method comprises: acquiring a monitoring image at a first moment, and on the basis of the monitoring image, acquiring a first element group and a second element group, wherein each of the element groups comprises position information of each element, and first elements and second elements are two types of elements of people in the monitoring image; and then on the basis of the position information, performing matching on the first elements and the second elements to acquire a plurality of element pairs, and on the basis of the at least one element pair, acquiring monitoring data in the monitoring image. Thus, monitoring data can be acquired on the basis of element pairs involving two types of elements, and thus the accuracy of the acquired monitoring data can be improved, thereby realizing the effect of improving the accuracy of the monitoring image processing method.

Description

Monitoring image processing method, device and non-volatile computer storage medium

[0001] This application claims priority to Chinese patent application number 202410317571.1, filed on March 19, 2024, entitled “Monitoring image processing method, device and non-volatile computer storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of monitoring technology, and in particular to a monitoring image processing method, device, and non-volatile computer storage medium. Background Art

[0003] Surveillance image processing technology is a technology that performs relevant analysis on surveillance images to obtain some data.

[0004] In a current surveillance image processing method, an image of the surveillance area is first acquired, and then a preset algorithm is used to obtain human body frames in the image. Each human body frame can be used to identify a person in the surveillance area. Then, based on the number of human body frames, some surveillance information such as the number of people in the surveillance area is obtained.

[0005] However, the above method of obtaining monitoring data by relying on human body detection has a low accuracy of the monitoring data obtained, which may result in a low accuracy of the monitoring image processing method. Summary of the Invention

[0006] The embodiments of the present application provide a monitoring image processing method, apparatus, and non-volatile computer storage medium. The technical solution is as follows:

[0007] According to a first aspect of the present application, a monitoring image processing method is provided, the method comprising:

[0008] Acquire a monitoring image of the monitoring area at the first moment;

[0009] Acquire, based on the surveillance image, a first element group and a second element group in the surveillance image, wherein the first element group includes position information of at least one first element, and the second element group includes position information of at least one second element, the first element and the second element being two elements of a person in the surveillance image;

[0010] Pairing the first element and the second element based on the position information of the first element and the position information of the second element to obtain at least one element pair, where the element pair includes a first element and a second element that are associated with each other;

[0011] Based on the at least one element pair, monitoring data in the monitoring image is determined.

[0012] Optionally, the first element group includes position information of a plurality of first elements, and the second element group includes position information of a plurality of second elements;

[0013] The pairing the first element and the second element based on the position information of the first element and the position information of the second element to obtain at least one element pair includes:

[0014] Selecting a first element to be matched from the multiple first elements;

[0015] Determine whether there is an associated second element with the first element to be matched;

[0016] When there is no associated second element for the first element to be matched, selecting a second element to be matched from the multiple second elements;

[0017] Determining whether the first element to be matched matches the second element to be matched based on the position information;

[0018] During matching, determining that the first element to be matched and the second element to be matched are an element pair, and executing the step of selecting a first element to be matched from the plurality of first elements;

[0019] When there is no match, the step of selecting a second element to be matched from the multiple second elements is performed.

[0020] Optionally, the method further includes:

[0021] When the first element has an associated second element, determining whether the first element to be matched matches the associated second element based on the position information;

[0022] During matching, determining that the first element to be matched and the associated second element are still an element pair;

[0023] When there is no match, it is determined that the first element to be matched and the associated second element are not an element pair, and based on the position information, the associated second element is paired with other first elements in the first element group except the paired first element.

[0024] Optionally, the second element is a human face, and after obtaining the first element group and the second element group in the surveillance image based on the surveillance image, the method further includes:

[0025] Acquire a tracking list and a lost tracking list based on the second element group, the tracking list including data of faces appearing in surveillance images at the first moment, and the lost tracking list including data of faces that appeared in surveillance images at a historical moment and did not appear in surveillance images during a preset time period starting from the historical moment, where the historical moment is a moment before the first moment;

[0026] The method further comprises:

[0027] When the first element has an associated second element, determining whether the associated second element is located in the tracking list;

[0028] When located in the tracking list, determining whether the first element to be matched matches the associated second element based on the position information;

[0029] During matching, determining that the first element to be matched and the associated second element are still an element pair;

[0030] When there is no match, determining that the first element to be matched and the associated second element are not an element pair, and pairing the associated second element with other first elements in the first element group except the paired first element based on the position information;

[0031] determining whether the associated second element is in the lost tracking list when it is not in the tracking list;

[0032] When the second element is located in the lost tracking list, performing the step of selecting a second element to be matched from the plurality of second elements;

[0033] When the first element to be matched and the associated second element are not in the lost tracking list, it is determined that the first element to be matched and the associated second element are still an element pair.

[0034] Optionally, the second element is a human face, and before determining whether the first element to be matched matches the second element to be matched based on the position information, the method further includes:

[0035] Determining whether the face quality of the image to be matched with the second element is greater than a preset face quality threshold;

[0036] When the face quality is greater than the preset face quality threshold, the step of determining whether the first element to be matched matches the second element to be matched based on the position information is performed.

[0037] Optionally, determining whether the first element to be matched matches the second element to be matched based on the position information includes:

[0038] Obtaining an overlapping area between an area occupied by the first element to be matched and an area occupied by the second element to be matched in the surveillance image;

[0039] Whether the first element to be matched matches the second element to be matched is determined based on the size of the overlapping area.

[0040] Optionally, the first element is a human head, and the second element is a human face, and determining whether the first element to be matched matches the second element to be matched based on the size of the overlapping area includes:

[0041] Determining an area ratio of the overlapping area to an area occupied by the first element to be matched in the monitoring image;

[0042] When the area ratio is greater than or equal to a preset ratio, determining that the first element to be matched matches the second element to be matched;

[0043] When the area ratio is smaller than the preset ratio, it is determined that the first element to be matched does not match the second element to be matched.

[0044] Optionally, selecting a first element to be matched from the multiple first elements includes:

[0045] Selecting, from the multiple first elements, a first element to be matched that occupies the largest area in the surveillance image and has not been selected in the current matching, wherein the current matching is a matching between the first element and the second element based on the surveillance image at the first moment;

[0046] When there is no associated second element for the first element to be matched, selecting a second element to be matched from the plurality of second elements includes:

[0047] When there is no associated second element with the first element to be matched, the second element to be matched that occupies the largest area in the monitoring image and has not been selected in this matching is selected from the multiple second elements.

[0048] Optionally, acquiring the first element group and the second element group in the surveillance image based on the image includes:

[0049] Acquire images of a plurality of first elements and images of a plurality of second elements in the surveillance image;

[0050] Arranging the plurality of first elements from largest to smallest in accordance with their sizes in the monitoring image to obtain a first element column;

[0051] Obtaining the first element group based on position information of first n first elements in the first element column, where n is an integer greater than zero;

[0052] Arranging the plurality of second elements in descending order of size to obtain a second element column;

[0053] The second element group is obtained based on position information of the first n second elements in the second element column.

[0054] Optionally, the second element is a human face, and before arranging the plurality of second elements in descending order of size to obtain a second element column, the method further includes:

[0055] The second elements whose image has a face quality less than a preset face quality threshold are removed from the multiple second elements.

[0056] Optionally, the second element is a human face, and determining the surveillance data in the surveillance image based on the at least one element pair includes:

[0057] Based on the image of the second element in the element pair, facial attribute information of the second element is obtained.

[0058] Optionally, determining the monitoring data in the monitoring image based on the at least one element pair includes:

[0059] The number of people in the surveillance image is determined according to the number of the at least one element pair.

[0060] According to another aspect of an embodiment of the present application, a monitoring device is provided, comprising:

[0061] An image acquisition module, configured to acquire a monitoring image of the monitoring area at a first moment;

[0062] an element group acquisition module, configured to acquire, based on the surveillance image, a first element group and a second element group in the surveillance image, wherein the first element group includes position information of at least one first element, the second element group includes position information of at least one second element, and the first element and the second element are two elements of a person in the surveillance image;

[0063] a pairing module, configured to pair the first element and the second element based on the position information of the first element and the position information of the second element, to obtain at least one element pair, the element pair including a first element and a second element that are associated with each other;

[0064] A data determination module is used to determine the monitoring data in the monitoring image based on the at least one element pair.

[0065] According to another aspect of an embodiment of the present application, a monitoring image processing device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the monitoring image processing method as described above.

[0066] According to another aspect of an embodiment of the present application, a computer storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the monitoring image processing method as described above.

[0067] A computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the surveillance image processing method provided in the various optional implementations described above.

[0068] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0069] By acquiring a surveillance image at a first moment, and based on the surveillance image obtaining a first element group and a second element group, wherein the element groups include position information of each element, the first element and the second element being two elements of a person in the surveillance image; then, the first element and the second element can be matched based on the position information to obtain multiple element pairs, and surveillance data from the surveillance image can be obtained based on at least one of the element pairs. In this way, surveillance data can be obtained based on the element pairs of the two elements, which can improve the accuracy of the obtained surveillance data, thereby achieving the effect of improving the accuracy of the surveillance image processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0071] FIG1 is a schematic diagram of an application scenario of a monitoring image processing system according to an embodiment of the present application;

[0072] FIG2 is a flow chart of a monitoring image processing method provided by an embodiment of the present application;

[0073] FIG3 is a flowchart of another monitoring image processing method provided by an embodiment of the present application;

[0074] FIG4 is a flow chart of a method for obtaining a first element group and a second element group in an embodiment of the present application;

[0075] FIG5 is a schematic diagram of matching between a first element and a second element shown in an embodiment of the present application;

[0076] FIG6 is a schematic diagram of an identification frame of a first element and an identification frame of a second element in an embodiment of the present application;

[0077] FIG7 is a schematic diagram of another identification frame of a first element and an identification frame of a second element in an embodiment of the present application;

[0078] FIG8 is a flowchart of another monitoring image processing method provided by an embodiment of the present application;

[0079] FIG9 is a flowchart of another monitoring image processing method provided by an embodiment of the present application;

[0080] FIG10 is a flowchart of determining an indicator mark according to an embodiment of the present application;

[0081] FIG11 is a block diagram of a monitoring device provided in an embodiment of the present application.

[0082] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0083] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0084] FIG1 is a schematic diagram of an application scenario of a monitoring image processing system shown in an embodiment of the present application. The application scenario may include a monitoring area q1 and a monitoring device. The monitoring device includes a camera 11 and a control component 12 .

[0085] The camera 11 is disposed near the monitoring area q1 and is used to capture images of the monitoring area q1.

[0086] A wired or wireless connection can be established between the control component 12 and the camera 11. The control component can obtain images captured by the camera 11 and perform operations such as image analysis and processing on the images. The control component 12 can be deployed in the same location as the camera 11, or in a different location from the camera 11, and this is not limited in the present embodiment. The control component 12 can be any device capable of image data processing, such as a computer, server, or server cluster.

[0087] Of course, in an exemplary embodiment, the control component 12 may not be directly connected to the camera 11 , and the control component 12 may obtain images from other devices and detect the images.

[0088] FIG2 is a flowchart of a monitoring image processing method provided by an embodiment of the present application. The method can be applied to the control component in the application scenario shown in FIG1 . The method includes the following steps:

[0089] Step 201: Acquire a monitoring image of a monitoring area at a first moment.

[0090] Step 202: Based on the surveillance image, a first element group and a second element group are obtained from the surveillance image. The first element group includes location information of at least one first element, and the second element group includes location information of at least one second element, wherein the first element and the second element are two elements of a person in the surveillance image.

[0091] Step 203: Pair the first element and the second element based on the position information of the first element and the position information of the second element to obtain at least one element pair, where the element pair includes a first element and a second element that are associated with each other.

[0092] Step 204: Determine monitoring data in the monitoring image based on at least one element pair.

[0093] In summary, the surveillance image processing method provided in the embodiments of the present application obtains a surveillance image at a first moment and, based on the surveillance image, obtains a first element group and a second element group. These element groups include position information of each element, and the first element and the second element are two elements of the person in the surveillance image. The first element and the second element can then be matched based on the position information to obtain multiple element pairs, and the surveillance data in the surveillance image can be confirmed based on the number of element pairs. In this way, surveillance data can be obtained based on the element pairs of the two elements, which can improve the accuracy of the obtained surveillance data, thereby achieving the effect of improving the accuracy of the surveillance image processing method.

[0094] FIG3 is a flowchart of another monitoring image processing method provided by an embodiment of the present application. The method can be applied to the control component in the application scenario shown in FIG1 . The method includes the following steps:

[0095] Step 301: Acquire a monitoring image of a monitoring area at a first moment.

[0096] When applying the surveillance image processing method provided in an embodiment of the present application, the control component may obtain a surveillance image of the surveillance area at a first moment. The first moment may be the current moment, or a moment before the current moment. The surveillance image may be a frame of image, and the control component may obtain a video stream and obtain the frame of image from the video stream.

[0097] The control component can continuously obtain monitoring images provided by the camera after acquisition, and can also continuously obtain monitoring images provided by other components.

[0098] Step 302: Acquire a first element group and a second element group in the surveillance image.

[0099] The first element group includes position information of a plurality of first elements, and the second element group includes position information of a plurality of second elements, where the first element and the second element are two of the three elements of a head, a face, and a body in the surveillance image. The control component may obtain the position information of the head, face, and body in the surveillance image based on a preset algorithm (such as a head detection algorithm, a face detection algorithm, and a body detection algorithm, or may simultaneously obtain at least two of the head, face, and body using a single detection algorithm), and the position information may record the position of the head, face, and body in the surveillance image.

[0100] Optionally, the control component can obtain the first element group and the second element group in the monitoring image through a pre-trained model. Exemplarily, the control component can obtain the first element group and the second element group through one model, or the control component can obtain the first element group through one model and obtain the second element group through another model.

[0101] The position information of the first element and the position information of the second element can be represented by an identification frame. The identification frame of the first element can be used to identify the position of the first element in the surveillance image, and the identification frame of the second element can be used to identify the position of the second element in the surveillance image. The control component can obtain the information of these identification frames as the position information of the first element and the position information of the second element. For example, the position information of a head can include a head frame (i.e., an identification frame of the head), and the area occupied by the head frame is the area where the head is located; the position information of a face can include a face frame (i.e., an identification frame of the face), and the area occupied by the face frame is the area where the face is located; the position information of a human body can include a human body frame (i.e., an identification frame of the human body), and the area occupied by the human body frame is the area where the human body is located; for a person in the surveillance image, the size of the human body frame is larger than the size of the head frame, and the size of the head frame is larger than the size of the face frame. Among them, the identification box may include the position of the element in the monitoring image, and the control component may record the identification box in a variety of ways. For example, the control component may record the coordinates of the upper left corner of the identification box, the width and height of the identification box, or the control component may record the coordinates of the lower left corner and the upper right corner of the identification box, etc.

[0102] In an exemplary embodiment, please refer to FIG. 4 , which is a flow chart of a method for obtaining a first element group and a second element group in an embodiment of the present application. Step 302 may include the following sub-steps:

[0103] Sub-step 3021: Acquire images of multiple first elements and images of multiple second elements in the surveillance image.

[0104] Sub-step 3022: Arrange the multiple first elements from large to small according to their sizes in the monitoring image to obtain a first element column.

[0105] The control component may arrange the identified multiple first elements in descending order of size in the monitoring image to obtain a first element column.

[0106] Sub-step 3023: Obtain a first element group based on position information of the first n first elements in the first element column.

[0107] The control component can obtain a first element group based on the position information of the first n first elements in the first element column, where n is an integer greater than zero. In other words, the control component can retain only the n largest first elements and remove the remaining first elements. This is because smaller first elements contain less useful information and may be misjudged. Therefore, preliminary screening can be performed at this step to improve subsequent matching speed.

[0108] Sub-step 3024: Arrange the plurality of second elements in descending order of size to obtain a second element column.

[0109] Similarly, the control component may arrange the plurality of second elements in descending order of size to obtain a second element column.

[0110] Sub-step 3025: Obtain a second element group based on position information of the first n second elements in the second element column.

[0111] Similarly, the control component may obtain the second element group based on position information of the first n second elements in the second element column.

[0112] The value of n can be determined by the number of first elements and the number of second elements detected in the surveillance image. First, n is less than or equal to n1, where n1 is equal to the smaller of the first and second elements. Alternatively, n can be x*n1, where x is a preset value between 0 and 1. For example, if x = 0.9, then n = 0.9*n1. The value of x can be determined by the number of element pairs that the control component can process within the duration of a single frame without affecting the frame rate.

[0113] In addition, when the second element is a human face, before the above sub-step 3024, the control component may further include:

[0114] The second elements whose image has a face quality less than a preset face quality threshold are removed from the plurality of second elements.

[0115] Because low-quality faces contain less valid information in the second element, the control component can remove some faces with low quality to reduce the amount of data processed in subsequent steps and improve overall processing speed. Facial quality, or facial image quality, is data reflecting the quality of a captured facial image. This facial quality can be related to multiple factors, such as image clarity, the percentage of eyes occluded, the percentage of the left face occluded, the percentage of the right face occluded, the percentage of the mouth occluded, the percentage of the nose occluded, and the percentage of the chin occluded. These percentages can be negatively correlated with facial quality.

[0116] In an embodiment of the present application, a face quality model can be used to determine whether the face quality of the second element image is less than a preset face quality threshold. The face quality threshold can be a preset value, and the face quality threshold can be related to some preset objectives. For example, if facial attribute information (such as the age and gender of the person corresponding to the face) is to be obtained, the face quality threshold can be a threshold that can obtain such facial attribute information.

[0117] In addition, after obtaining the first element group and the second element group, the control component can track the first element in the first element group and the second element in the second element group respectively through a tracking algorithm, so as to facilitate confirmation of tracking the same first element and the same second element in surveillance images of different frames.

[0118] Step 303: Select a first element to be matched from multiple first elements.

[0119] After obtaining the first element group, the control component may select a first element to be matched from the plurality of first elements. The selection may be random or based on a preset method.

[0120] For example, one approach may include:

[0121] Select the first element to be matched from the multiple first elements that occupies the largest area in the surveillance image (for example, the multiple first elements can be sorted from largest to smallest in terms of area occupied and then selected) and that has not been selected in the current matching. This matching refers to the matching of the first element with the second element based on the surveillance image at the first moment.

[0122] That is, each time the control component executes this step, it can select the first element to be matched that occupies the largest area in the monitoring image from the first element group, and the selected first element will not be selected when step 303 is subsequently executed, so as to achieve the effect of traversing the first element group in order from large to small according to size.

[0123] Step 304 , determine whether there is an associated second element with the first element to be matched; if there is no associated second element with the first element to be matched, execute step 305 ; if there is an associated second element with the first element to be matched, execute step 309 .

[0124] The control component can determine whether the selected first element to be matched has an associated second element, that is, the control component can detect whether the first element to be matched has an association with a second element in the matching process corresponding to a certain moment before the first moment (this matching process can be the matching process corresponding to the first moment).

[0125] After each element pair between a first element and a second element is established, the control component can record the data of the element pair in a local or cloud-based storage device. Based on the data stored in the storage device, the control component can then determine whether there is an associated second element with the first element to be matched. Each element pair can correspond to a person in the monitored area. When the second element is a face, the control component can also associate an image of the face with the element pair. For example, the image of the face can also be stored in the element pair. This facilitates obtaining certain facial attribute information of the person based on the image of the face.

[0126] Step 305: Select a second element to be matched from the plurality of second elements and proceed to step 306.

[0127] The control component may select a second element to be matched from the plurality of second elements when there is no associated second element with the first element to be matched.

[0128] The control component may select the second element in a manner similar to step 303 above. For example, the control component may select the second element to be matched that occupies the largest area in the surveillance image and has not been selected in this matching from among multiple second elements.

[0129] That is, each time the control component executes this step, it can select the second element to be matched that occupies the largest area in the monitoring image from the second element group, and the selected second element will not be selected in the subsequent execution of step 305, so as to achieve the effect of traversing the second element group in descending order according to size. Of course,

[0130] At this point, the subsequent steps can be related steps for matching the selected first element to be matched and the second element to be matched. For example, please refer to Figure 5, which is a matching schematic diagram of a first element and a second element shown in an embodiment of the present application. Among them, multiple first elements and multiple second elements are arranged from large to small in size and matched accordingly. Since the first element and the second element are selected from large to small in size, the larger first element will be matched with the larger second element first, which will increase the success rate of the matching, thereby shortening the overall matching time and improving the implementation efficiency of the method provided in the embodiment of the present application.

[0131] Of course, this step can also be reversed to select the smallest second element. For example, the control component can select the second element to be matched from multiple second elements, which occupies the smallest area in the monitoring image and has not been selected in this matching. Correspondingly, the above step 303 will also become: select the first element to be matched from multiple first elements, which occupies the smallest area in the monitoring image and has not been selected in this matching.

[0132] Step 306: Determine whether the face quality of the image to be matched with the second element is greater than a preset face quality threshold. If so, execute step 307; if not, execute step 305.

[0133] Before matching the second element to be matched, the control component can determine whether the facial quality of the image of the second element to be matched is greater than a preset facial quality threshold, and only start matching the first element and the second element when it is greater. This can ensure that the image of the second element in the subsequently determined element pair has facial attribute information that can be collected (that is, the facial quality reaches a level that can obtain facial attribute information from the image in a preset manner).

[0134] In addition, for the content of the face quality threshold, please refer to the above sub-step 3025, and the embodiment of the present application will not be repeated here.

[0135] Furthermore, step 306 is optional. If the facial quality of the image after the second element has been removed in sub-step 3025 is less than the second element of the preset face quality threshold, then step 306 can be skipped and step 307 can be executed after step 305 is executed. Conversely, if the facial quality of the image after the second element has been removed in sub-step 3025 is less than the second element of the preset face quality threshold, then step 306 can be executed after step 305 is executed.

[0136] Furthermore, in step 306, upon determining that the face quality exceeds a preset face quality threshold, the control component may first determine whether the second element to be matched has an associated first element. If so, the control component may return to step 305 to reselect the second element to be matched. If not, the control component may proceed to step 307. This avoids matching an already associated second element with another first element. Furthermore, between steps 306 and 305, the control component may first determine whether the second element to be matched has an associated first element. If so, the control component may return to step 305 to reselect the second element to be matched. If not, the control component may proceed to step 306.

[0137] Step 307: Determine whether the first element to be matched matches the second element to be matched based on the position information. If they match, execute step 308; if they do not match, execute step 305.

[0138] Since the face and head of the same person are located at similar positions in the surveillance image, based on this, the control component can determine whether the two elements match through the position information of the first element to be matched and the position information of the second element to be matched.

[0139] In the method provided in the embodiment of the present application, it is possible to determine whether the first element to be matched matches the second element to be matched based on the position information in a variety of ways. For example, one way includes:

[0140] 1) Obtaining the overlapping area of ​​the area occupied by the first element to be matched and the area occupied by the second element to be matched in the monitoring image.

[0141] The area occupied by the first element to be matched can be the area surrounded by the identification box of the first element, and the area occupied by the second element to be matched can be the area surrounded by the identification box of the second element. Then, the overlapping area of ​​the area occupied by the first element to be matched and the area occupied by the second element to be matched is the intersection of the area surrounded by the identification box of the first element and the area surrounded by the identification box of the second element.

[0142] 2) Determine whether the first element to be matched matches the second element to be matched based on the size of the overlapping area.

[0143] The control component may determine whether the first element to be matched matches the second element to be matched based on the acquired overlapping area.

[0144] In a specific embodiment, the first element is a human head and the second element is a human face. The control component determines whether the first element to be matched matches the second element to be matched in a manner that may include:

[0145] 2.1) Determine the ratio of the overlapping area to the area occupied by the first element to be matched in the monitoring image.

[0146] The control component may first determine an area ratio between the overlapping area and the area occupied by the first element to be matched (human head) in the monitoring image.

[0147] For example, please refer to Figure 6, which is a schematic diagram of an identification frame of a first element and an identification frame of a second element in an embodiment of the present application, wherein there is an overlapping area k3 between the identification frame k1 of the first element and the identification frame k2 of the second element, then the control component can obtain the ratio of the area of ​​the overlapping area k3 to the area of ​​the area surrounded by the identification frame k1 of the first element. This ratio can reflect the degree of proximity between the position of the first element to be matched and the position of the second element to be matched in the surveillance image. The larger the area ratio, the closer the position of the first element to be matched is to the position of the second element to be matched. The control component can determine whether the first element to be matched and the second element to be matched match based on the area ratio. This area ratio can be called the degree of overlap (Intersection over Union, IoU).

[0148] Optionally, in Figure 6, the coordinates of the vertex of the upper left corner of the identification box k1 of the first element are (x11, y11), and the coordinates of the vertex of the lower right corner are (x12, y12). Then the horizontal width of the identification box k1 of the first element in Figure 6 is w1 = x12-x11, and the height is h1 = y12-y11; the coordinates of the vertex of the upper left corner of the identification box k2 of the second element are (x21, y21), and the coordinates of the vertex of the lower right corner are (x22, y22). The horizontal width of the identification box k2 of the second element in Figure 6 is w2 = x22-x21, and the height is h2 = y22-y21; then the horizontal width of the overlapping area k3 in Figure 6 is w = w2+w1-(x22-x11), and the height is h = h2+h1-(y22-y11). In this way, the area of ​​the overlapping area k3 can be calculated.

[0149] 2.2) When the area ratio is greater than or equal to the preset ratio, it is determined that the first element to be matched matches the second element to be matched.

[0150] When the area ratio is greater than or equal to a preset ratio, it indicates that the position of the first element to be matched is very close to the position of the second element to be matched. In this case, the control component can determine that the first element to be matched matches the second element to be matched. In other words, the head of the first element and the face of the second element are the same person. The preset ratio value can be related to the scene in which the monitoring area is located. For example, the preset ratio can be different for indoor and outdoor scenes. For example, the preset ratio can be 0.3.

[0151] In addition, when the first element is a human body and the second element is a head or a face, after determining that the area ratio is greater than or equal to the preset ratio, the control component can also determine whether the second element is located in the upper area of ​​the area where the first element is located. When it is located in the upper area, it is determined that the first element to be matched matches the second element to be matched. When it is not located in the upper area, it is determined that the first element to be matched does not match the second element to be matched. This is because for a person, the head or face will be located in the upper part of the human body, and thus the accuracy of the method provided in the embodiment of the present application can be further improved in this way. Among them, the area where the first element in the monitoring image is located can be divided into n parts from top to bottom, and the x areas starting from the top can be the upper area, x is greater than or equal to 1 and less than n, and for example, n=3, x=1, or n=2, x=1, or n=4, x=1.

[0152] FIG6 shows a case where the area ratio is greater than the preset ratio. In this case, the control component can determine that the first element to be matched matches the second element to be matched.

[0153] In addition, since a human face is smaller than a human head, the control component can determine that the first element to be matched and the second element to be matched do not match when the area ratio is greater than 1, thereby reducing the possibility of misjudgment.

[0154] 2.3) When the area ratio is less than the preset ratio, it is determined that the first element to be matched does not match the second element to be matched.

[0155] When the area ratio is less than a preset ratio, it indicates that the position of the first element to be matched is relatively far from the position of the second element to be matched. In this case, the control component can determine that the first element to be matched matches the second element to be matched. In other words, the head of the first element and the face of the second element are the same person.

[0156] Please refer to Figure 7, which is a schematic diagram of another first element identification frame and a second element identification frame in an embodiment of the present application. Figure 7 shows a schematic diagram of the relationship between two first element identification frames and second element identification frames. In case A, there is an overlapping area k3 between the first element identification frame k1 and the second element identification frame k2, but the overlapping area k3 is obviously small, and the area ratio of the overlapping area k3 to the area occupied by the identification frame k1 of the first element to be matched is less than a preset ratio. In case B, there is no overlapping area between the first element identification frame k1 and the second element identification frame k2. In this case, the area of ​​the overlapping area is zero, and the area ratio of the area of ​​the overlapping area to the area occupied by the identification frame k1 of the first element to be matched is obviously less than a preset ratio. Figure 7 shows a case where the area ratio of the area of ​​the two overlapping areas to the area occupied by the identification frame k1 of the first element to be matched is less than a preset ratio. In both cases, the control component can determine that the first element to be matched and the second element to be matched do not match.

[0157] Correspondingly, when there is no match, the control component may execute step 305 to select the next second element for matching.

[0158] Step 308: Determine that the first element to be matched and the second element to be matched are an element pair, and execute step 303.

[0159] When the first element to be matched matches the second element to be matched, the control component can determine that the first element to be matched and the second element to be matched are an element pair. In this way, an element pair of the first element and the second element is determined. The control component can record the data of the element pair in a memory (local memory or cloud memory), and then determine whether there is an associated second element with the first element to be matched based on the data in the memory.

[0160] In addition, in addition to this matching method, in an embodiment of the present application, the control component can also determine the area ratio of each two elements (the two elements may refer to a first element and a second element) after obtaining the first element group and the second element group in step 302 (the area ratio refers to the ratio of the overlapping area of ​​the first element and the second element to the area occupied by the first element), and determine which two elements are an element pair based on the size of the area ratio.

[0161] Step 309: Determine whether the first element to be matched matches the associated second element based on the position information. If they match, execute step 310; if they do not match, execute step 311.

[0162] When a first element is associated with a second element, the control component can determine whether the first element to be matched matches the associated second element based on the position information. The associated relationship can be established by the control component based on a surveillance image at a moment before the first moment, that is, at a moment before the first moment, the control component can execute the method provided in the embodiment of the present application and, based on the method, determine that the first element and the second element are an element pair, and thus the first element and the second element are two mutually associated elements.

[0163] On this basis, in this step, the associated relationship established before the first moment can be verified to verify whether the associated relationship still exists in the monitoring image at the first moment. In this way, it is possible to verify in real time whether the first element and the second element in each element pair are still matched, so as to ensure the timeliness and accuracy of each determined element pair, realize the effect of real-time update of the element pairs, and thus improve the accuracy of some data determined based on the element pairs (such as the number of people in the monitoring area).

[0164] In this step, the process of the control component determining whether the first element to be matched and the associated second element match based on the position information can refer to the above-mentioned step 307, that is, the method of determining whether the first element to be matched and the second element to be matched match in step 307 can be used to determine whether the first element to be matched and the associated second element match. For example, in step 309, the second element to be matched in the determination method used in step 307 can be replaced by the second element associated with the first element to be matched.

[0165] Step 310: Determine that the first element to be matched and the associated second element are still an element pair.

[0166] When the first element to be matched matches the associated second element, the control component can determine that the first element to be matched and the associated second element are still an element pair, thereby completing the verification of the matching first element and the associated second element in this matching process.

[0167] Step 311: Determine that the first element to be matched and the associated second element are not an element pair, and pair the associated second element with other first elements in the first element group except the paired first element based on the position information.

[0168] When the first element to be matched does not match the associated second element, the control component can determine that the first element to be matched and the associated second element are not an element pair. At this time, it means that the association relationship established before the first moment no longer exists. Then, the control component can pair the associated second element with other first elements in the first element group except the paired first element based on the position information.

[0169] The reasons for this situation may be that when the control component tracks the first element and the second element, the tracking algorithm may cause the first element or the second element to be tracked incorrectly. For example, the first element a may be mistaken for the first element b, and then the first element a may be mistakenly believed to be associated with the second element b (the second element b is the second element associated with the first element b). Since IDs may be assigned to the first element and the second element during tracking, this problem is also called a string ID problem.

[0170] The method provided in the embodiment of the present application avoids the occurrence of string ID problems by pairing the first element with the second element and checking them in real time, thereby achieving the effect of error correction for incorrectly determined element pairs.

[0171] In the field of monitoring technology, it is difficult to avoid errors in some monitoring data. The embodiment of the present application uses this method to achieve real-time error correction of erroneous data, thereby improving the accuracy of the monitoring image processing method provided by the embodiment of the present application.

[0172] In addition, the method for pairing the associated second element with other first elements in the first element group except the paired first element may refer to other steps in the embodiments of the present application. For example, the pairing method may include:

[0173] 1) Reselect a first element to be matched from the first element group.

[0174] The control component may select a first element to be matched from other first elements in the first element group, excluding the paired first element. For example, if the first element group includes five first elements A1, A2, A3, A4, and A5, and the first element to be matched that is determined not to form an element pair with the associated second element is A1, then in this step, first element A2 may be selected as the new first element to be matched.

[0175] 2) Determine whether the associated second element matches the reselected first element to be matched.

[0176] The control component determines whether the associated second element matches the reselected first element to be matched based on the method of step 307. When they match, the step of pairing the associated second element with other first elements in the first element group except the paired first element is completed. When they do not match, the above 1) can be re-executed.

[0177] After re-pairing the associated second element with other first elements in the first element group except the paired first element, a new element pair is obtained. When the second element is a human face, the control component can update the image of the human face included in the element pair based on the image of the associated second element in the surveillance image at the first moment, so as to realize the update of the image of the human face and ensure the timeliness and accuracy of the image of the human face in the element pair.

[0178] If the pairing is successful, indicating that the associated second element and the new first element form an element pair, the control component can then update the information of the element pair based on the information of the associated second element. For example, if the associated second element is Face A and the new first element is Head A, the control component can update the face information corresponding to Head A in the database with the information of Face A. The control component can then execute step 305 to select the next second element and continue the matching process for the first element to be matched. If the pairing fails, indicating that there is currently no first element that matches the associated second element, step 305 can also be executed in this case.

[0179] Step 312: After traversing the first element in the first element group, determine the monitoring data in the monitoring image based on at least one element pair.

[0180] In step 303, a first element to be matched in the first element group is selected, and the first element to be matched is processed by the method provided in the embodiment of the present application, and three processing results of step 308, step 310 or step 311 may be obtained. If the control component selects the last first element to be matched in the first element group in step 303 and processes the last first element to be matched until step 308, step 310 or step 311, it is considered that the traversal of the first element in the first element group is completed. At this time, this step can be executed to determine the monitoring data in the monitoring image based on at least one element pair.

[0181] The monitoring data may include the number of people in the monitoring area and other monitoring data. For example, the control component may determine the number of people in the monitoring area based on the number of element pairs. For example, the number of element pairs may be determined as the number of people in the monitoring area. That is, only when two elements of an object in the monitoring area are detected is the object determined to be a person. This can improve the accuracy of the determined number of people.

[0182] In the related art, when face tracking technology is used alone to monitor people in a monitoring area, due to the face tracking algorithm, after some faces are lost and reappear, there may be a situation where one person corresponds to multiple face tracking identifiers. If multiple tracking identifiers are confirmed as multiple people, it will lead to a problem of low accuracy of the monitoring data. In the method provided in the embodiment of the present application, the accuracy of the monitoring data can be improved by associating the head and the face as an element pair.

[0183] In an exemplary embodiment, the control component may further obtain facial attribute information of the second element in the element pair based on the image of the second element. This facial attribute information may include information such as the gender, age, and facial expression of the person corresponding to the face. Based on this information, the control component may generate statistics such as the overall gender ratio and age distribution of people in the monitored area.

[0184] In summary, the surveillance image processing method provided in the embodiments of the present application obtains a surveillance image at a first moment and, based on the surveillance image, obtains a first element group and a second element group. These element groups include position information of each element, and the first element and the second element are two elements of the person in the surveillance image. The first element and the second element can then be matched based on the position information to obtain multiple element pairs, and the surveillance data in the surveillance image can be confirmed based on the number of element pairs. In this way, surveillance data can be obtained based on the element pairs of the two elements, which can improve the accuracy of the obtained surveillance data, thereby achieving the effect of improving the accuracy of the surveillance image processing method.

[0185] FIG8 is a flowchart of another monitoring image processing method provided by an embodiment of the present application, which can be applied to the control component in the monitoring device shown in FIG1. ​​The method may include the following steps:

[0186] Step 801: Acquire a monitoring image of a monitoring area at the first moment.

[0187] This step can refer to step 301 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0188] Step 802: Based on the monitoring image, obtain a first element group and a second element group in the monitoring area.

[0189] The first element group includes position information of multiple first elements, the second element group includes position information of multiple second elements, and the first element and the second element are two of the three elements of the head, face and body of the person in the monitoring image.

[0190] This step can refer to step 302 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0191] Step 803: Obtain a tracking list and a lost tracking list based on the second element group.

[0192] In the embodiment of the present application, the first element is a human head (or human body) and the second element is a human face. Based on this, the tracking list includes data on faces that appear in surveillance images at a first moment, and the lost tracking list includes data on faces that appeared in surveillance images at a historical moment and did not appear in surveillance images for a preset period of time starting from the historical moment, where the historical moment is a moment before the first moment.

[0193] The control component can use a face detection algorithm to continuously identify multiple surveillance images of the monitoring area (for example, each frame of the continuously collected image) and track the identified faces to obtain the above-mentioned tracking list and lost tracking list. Exemplarily, the control component can record the data of the faces appearing in the surveillance images of the monitoring area in the tracking list, and track each face in the tracking list (for example, a tracking identifier (ID) can be assigned to each detected face, and the position of the face can be marked with an identification box, and the tracking identifier and position can be recorded in the tracking list). When a face leaves the monitoring area at a certain moment (that is, the face is not detected in the surveillance image at that moment), the control component removes the face from the tracking list and starts timing. When the preset time is reached, the face information can be adjusted to the lost tracking list, so that the above-mentioned setting of the tracking list and the lost tracking list can be achieved. For example, when a person appears in the monitoring area for the first time, the control component can detect the face frame of the person, and then track the face frame through the tracking algorithm. At this time, the control component puts the face information (such as the face frame and the face tracking identifier) ​​into the tracking list. From the time the person appears to the time he walks out of the monitoring area, the face information is always in the tracking list until the person walks out of the monitoring range. When the face information does not appear in the tracking list for a period of time (preset duration), the control component can put the face information into the lost tracking list, and then obtain the face tracking identifier and face frame in the lost tracking list.

[0194] By using this recording method, it is possible to avoid the situation where part of a face is mistakenly counted as a new face when it is re-detected after being lost for a short time, thereby improving the accuracy of the method provided in the embodiment of the present application. For example, if it is set that the face is lost for half a second, it is considered as a tracking loss, then the preset duration can also be set to a different number of frames according to the frame rate of the video stream obtained by the control component. For example, if the frame rate of the video stream obtained by the control component is 24 Hz, the preset duration can be about 12 frames (for example, 10 to 14 frames). Alternatively, the preset duration can also be about 3 frames (for example, 2 to 4 frames).

[0195] The control component may use a face tracking algorithm to track the face, for example, a Simple Online and Realtime Tracking (SORT) algorithm may be used to track the face.

[0196] In addition, regarding the tracking list and the lost tracking list in the face tracking technology, reference may be made to related technologies, and the embodiments of the present application do not limit this.

[0197] Step 804: Select a first element to be matched from multiple first elements.

[0198] This step can refer to step 303 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0199] Step 805 , determine whether there is an associated second element with the first element to be matched; if there is no associated second element with the first element to be matched, execute step 806 ; if there is an associated second element with the first element to be matched, execute step 810 .

[0200] This step can refer to step 304 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0201] Step 806: Select a second element to be matched from the multiple second elements. Execute step 807.

[0202] When there is no associated second element for the first element to be matched, the control component may select a second element to be matched from multiple second elements.

[0203] This step can refer to step 305 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0204] Step 807: Determine whether the face quality of the image to be matched with the second element is greater than a preset face quality threshold. If so, execute step 808; if not, execute step 806.

[0205] This step can refer to step 306 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0206] Step 808: Determine whether the first element to be matched matches the second element to be matched based on the position information. If they match, execute step 809; if they do not match, execute step 806.

[0207] This step can refer to step 307 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0208] Step 809 : Determine that the first element to be matched and the second element to be matched are an element pair, and execute step 804 .

[0209] This step can refer to step 308 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0210] Step 810: Determine whether the associated second element is in the tracking list. If it is in the tracking list, execute step 811; if not, execute step 814.

[0211] When a first element is associated with a second element, the control component may determine whether the associated second element is in the tracking list. Exemplarily, the control component may determine that the associated second element is in the tracking list when a tracking identifier of the associated second element is in the tracking list, and correspondingly, determine that the associated second element is not in the tracking list when the tracking identifier of the associated second element is not in the tracking list.

[0212] The second element recorded in the tracking list is the face of a person in the surveillance image captured at the first moment (the first moment may be the current moment). The control component can determine whether the person corresponding to the second element is the face appearing in the surveillance image at the first moment by determining whether the associated second element is in the tracking list. When the associated second element is in the tracking list, it indicates that the associated second element is the face appearing in the surveillance image at the first moment. When the associated second element is not in the tracking list, it can be further determined whether the associated second element is in the lost tracking list.

[0213] Step 811: Determine whether the first element to be matched matches the associated second element based on the position information; if they match, execute step 812; if they do not match, execute step 813.

[0214] When located in the tracking list, the control component can determine whether the first element to be matched matches the associated second element based on the position information. The specific method of determining whether the first element to be matched matches the associated second element based on the position information can refer to step 309 in the above embodiment, and the embodiments of this application will not be repeated here.

[0215] Step 812: Determine that the first element to be matched and the associated second element are still an element pair.

[0216] When the first element to be matched matches the associated second element, the control component can determine that the first element to be matched and the associated second element are still an element pair. This step can refer to step 310 in the above embodiment, and will not be repeated in this embodiment of the present application.

[0217] Step 813: Determine that the first element to be matched and the associated second element are not an element pair, and pair the associated second element with other first elements in the first element group except the paired first element based on the position information.

[0218] When there is no match, the control component may determine that the first element to be matched and the associated second element are not an element pair, and based on the position information, pair the associated second element with other first elements in the first element group except the paired first element. This step can refer to step 311 in the above embodiment, and will not be repeated in this embodiment of the present application.

[0219] After this step is completed, step 806 may be executed to perform a matching process on the first element to be matched.

[0220] Step 814: Determine whether the associated second element is in the loss tracking list. If it is in the loss tracking list, execute step 806; if it is not in the loss tracking list, execute step 804.

[0221] When the associated second element is not in the tracking list, it indicates that the control component did not detect the associated second element in the surveillance image captured at the first moment. In this case, the control component may continue to determine whether the associated second element is in the lost tracking list. Exemplarily, the control component may determine that the associated second element is in the lost tracking list when the tracking identifier of the associated second element is in the lost tracking list, and correspondingly, determine that the associated second element is not in the lost tracking list when the tracking identifier of the associated second element is not in the lost tracking list.

[0222] If the associated second element is in the lost tracking list, it indicates that the associated second element is the second element that has been lost for a long time (preset duration), and the control component can consider that the association between the first element to be matched and the associated second element is invalid. At this time, step 806 can be executed to start the re-matching process.

[0223] If the associated second element is not in the lost tracking list, it indicates that the associated second element has not been lost for a long time. At this time, it can be considered that the first element to be matched is still associated with the associated second element, and step 804 is executed to start matching the next first element.

[0224] Step 815: After traversing the first element in the first element group, determine the monitoring data in the monitoring image based on at least one element pair.

[0225] In step 804, a first element to be matched in the first element group is selected, and the first element to be matched is processed by the method provided in the embodiment of the present application, and four processing results may be obtained, namely step 809, step 812, step 813 or step 814. If the control component selects the last first element to be matched in the first element group in step 804 and processes the last first element to be matched until step 809, step 812 or step 814, it is considered that the traversal of the first element in the first element group is completed. At this time, this step can be executed to determine the monitoring data in the monitoring image based on at least one element pair.

[0226] The monitoring data may include the number of people in the monitoring area and other monitoring data. For example, the control component may determine the number of people in the monitoring area based on the number of element pairs. For example, the number of element pairs may be determined as the number of people in the monitoring area. That is, only when two elements of an object in the monitoring area are detected is the object determined to be a person. This can improve the accuracy of the determined number of people.

[0227] In addition, in the method provided in the embodiment of the present application, the control component can also obtain facial attribute information at other times. For example, the control component can obtain facial attribute information in real time after obtaining the image of the face, or the control component can obtain facial attribute information based on the collected facial image after the person corresponding to the face leaves the monitoring area (that is, after the face is lost from the tracking list).

[0228] In addition, this step can also refer to step 312 in the above embodiment, or step 312 can also refer to this step, and the embodiments of this application will not be repeated here.

[0229] In summary, the surveillance image processing method provided in the embodiments of the present application obtains a surveillance image at a first moment and, based on the surveillance image, obtains a first element group and a second element group. These element groups include position information of each element, and the first element and the second element are two elements of the person in the surveillance image. The first element and the second element can then be matched based on the position information to obtain multiple element pairs, and the surveillance data in the surveillance image can be confirmed based on the number of element pairs. In this way, surveillance data can be obtained based on the element pairs of the two elements, which can improve the accuracy of the obtained surveillance data, thereby achieving the effect of improving the accuracy of the surveillance image processing method.

[0230] FIG9 is a flowchart of another monitoring image processing method provided by an embodiment of the present application, which can be applied to the control component in the monitoring device shown in FIG1. ​​The method may include the following steps:

[0231] Step 901: Acquire a monitoring image of a monitoring area at the first moment.

[0232] This step can refer to step 301 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0233] Step 902: Based on the monitoring image, obtain a first element group and a second element group in the monitoring area.

[0234] The first element group includes position information of multiple first elements, the second element group includes position information of multiple second elements, and the first element and the second element are two of the three elements of the head, face and body of the person in the surveillance image.

[0235] In addition, in step 902, the control component may also obtain a tracking list and a lost tracking list based on the second element group. The specific method may refer to step 803 in the above embodiment, and will not be described in detail in this embodiment.

[0236] Step 903: Select a first element to be matched from multiple first elements.

[0237] This step can refer to step 303 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0238] Step 904: Determine the indicator of the associated information of the first element to be matched.

[0239] The indicator identifier can be used to indicate the association status of the first element, and the association status may include associated (i.e., the first element has an associated second element), unassociated (i.e., the first element does not have an associated second element), and association error (i.e., the second element previously associated with the first element no longer exists in the monitoring image at the first moment, which corresponds to the situation in step 813 in the above embodiment). When associated, the control component can determine the indicator identifier as the first identifier; when an association error occurs, the control component can determine the indicator identifier as the second identifier; when unassociated, the control component can determine the indicator identifier as the third identifier. Exemplarily, the first identifier is 1, the second identifier is -1, and the third identifier is 0.

[0240] The control component may determine the indicator in a preset manner. For example, please refer to FIG10 , which is a flow chart of determining an indicator provided in an embodiment of the present application. The flow chart may include:

[0241] Step 9041: Determine whether there is an associated second element with the first element to be matched; if there is no associated second element with the first element to be matched, execute step 9042; if there is an associated second element with the first element to be matched, execute step 9043.

[0242] This step can refer to step 304 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0243] Step 9042: Determine that the indication identifier is the third identifier.

[0244] When there is no associated second element with the first element to be matched, the association status of the first element is unassociated. In this case, the control component may determine that the indication identifier is the third identifier.

[0245] Step 9043: Determine whether the associated second element is in the tracking list. If it is in the tracking list, execute step 9044; if it is not in the tracking list, execute step 9047.

[0246] The control component can obtain the identifier of the associated second element and determine whether the associated second element is in the tracking list based on the identifier. This step can also refer to step 810 in the above embodiment, and the embodiment of this application will not be repeated here.

[0247] Step 9044: Determine whether the first element to be matched matches the associated second element based on the position information; if they match, execute step 9045; if they do not match, execute step 9046.

[0248] This step can refer to step 811 in the above embodiment, and will not be repeated here in this embodiment of the present application.

[0249] Step 9045: Determine that the indication identifier is the first identifier.

[0250] When it is determined that the first element to be matched matches the associated second element, it indicates that the first element to be matched and the associated second element are still an element pair, and the association relationship still exists, and then the control component can determine that the indicator identifier is the first identifier.

[0251] Some descriptions of this step can also refer to step 310 in the above embodiment, and will not be repeated here in this embodiment of the present application.

[0252] Step 9046: Determine that the indication identifier is the second identifier.

[0253] When it is determined based on the position information that the first element to be matched and the associated second element do not match, it indicates that the first element to be matched and the associated second element are no longer an element pair, and the association relationship between the first element to be matched and the associated second element is wrong. At this time, the control component can determine that the indicator identifier is the second identifier.

[0254] Step 9047: Determine whether the associated second element is in the loss tracking list. If it is in the loss tracking list, execute step 9048; if it is not in the loss tracking list, execute step 9045.

[0255] If the first element to be matched is not in the lost tracking list, it indicates that the associated second element has not been completely lost. In this case, it can be considered that the first element to be matched and the associated second element still match, and step 9045 is executed.

[0256] In addition, this step can refer to step 814 in the above embodiment, and the embodiment of the present application will not be repeated here.

[0257] Step 9048: Add the data of the second element to the data of the first element. Execute step 9042.

[0258] When the associated second element is in the lost tracking list, the control component can add the information of the second element to the information of the first element. Adding the information of the second element to the information of the first element can mean that the control component assigns a data group to the first element. In this step, the data of the second element can be recorded in the data of the first element. This is because when the associated second element is in the lost tracking list, it indicates that the second element has lost tracking. In order to avoid completely losing the data of the second element, the control component can add the data of the second element to the data of the first element. Later, when the second element appears again in the monitoring area, the data of the second element added to the data of the first element can be used. In an exemplary embodiment, the second element is a human face and the first element is a human head. The data of the second element includes image information of the face, a tracking identifier of the face, and information such as a recognition frame of the face.

[0259] In addition, when the associated second element is in the lost tracking list, the second element has been lost. At this time, it can be considered that there is no associated second element for the first element to be matched, and then step 9042 can be executed.

[0260] Through the above sub-steps 9041 to 9048, it can be determined whether the indicator mark is the first mark, the second mark or the third mark.

[0261] Step 905: Determine whether the indicator is the first indicator. If the indicator is the first indicator, execute step 903; if the indicator is not the first indicator, execute step 906.

[0262] After determining the indication in step 904 , the control component may first determine whether the indication identifier is the first identifier.

[0263] Step 906: Determine whether the indicator is the second identifier. If the indicator is the second identifier, execute step 907; if the indicator is not the second identifier, execute step 910.

[0264] When the indication identifier is not the first identifier, the control component may continue to determine whether the indication identifier is the second identifier.

[0265] Step 907: Pair the second element with other first elements in the first element group except the paired first elements.

[0266] When the indication identifier is the second identifier, it indicates that the current situation is an association error. At this time, the control component can pair the second element with other first elements in the first element group except the paired first element based on the position information.

[0267] This step can also refer to step 311 in the above embodiment, and will not be repeated here in this embodiment of the present application.

[0268] Step 908: Determine whether the pairing is successful. If successful, execute step 909; if unsuccessful, execute step 910.

[0269] The control component may determine whether the pairing in step 907 is successful.

[0270] Step 909: Update the data of the second element and proceed to step 910.

[0271] If the pairing is successful, it indicates that the associated second element and the new first element form an element pair. Updating the second element's data may refer to the control component updating the information of the element pair based on the information of the associated second element. For example, if the associated second element is face A and the new first element is head A, the control component may update the face information corresponding to head A in the database with the information of face A. The control component may then execute step 911 to select the next second element and continue the matching process for the first element to be matched.

[0272] Step 910: Select a second element to be matched from multiple second elements.

[0273] When the indicator identifier is neither the first identifier nor the second identifier, it indicates that the indicator identifier is the third identifier, that is, the association status of the first element is unassociated. At this time, a second element to be matched can be selected from multiple second elements to start the process of matching the first element.

[0274] This step can refer to step 305 in the above embodiment, and will not be described in detail in this embodiment of the present application.

[0275] Step 911: Determine whether the face quality of the image to be matched with the second element is greater than a preset face quality threshold. If so, execute step 912; if not, execute step 910.

[0276] The embodiment of the present application is described by taking the second element being a human face as an example. This step can refer to step 306 in the above embodiment, and the embodiment of the present application will not be repeated here.

[0277] Step 912: Determine whether the second element to be matched is associated with the first element. If the second element to be matched is associated with the first element, execute step 910; if the second element to be matched is not associated with the first element, execute step 913.

[0278] The control component may determine whether the selected second element to be matched is a second element that has been associated with the first element. If so, the control component may return to step 910 to reselect the second element. If not, the control component may proceed to subsequent step 913.

[0279] In addition, in step 910 , when selecting the second element, a second element to be matched that is not associated with the first element may be selected, and this embodiment of the present application does not limit this.

[0280] Step 913: Determine whether the first element to be matched matches the second element to be matched. If they match, execute step 914; if they do not match, execute step 910.

[0281] The control component can determine whether the first element to be matched matches the second element to be matched based on the position information. This step can also refer to step 307 in the above embodiment, and the embodiment of the present application will not be repeated here.

[0282] Step 914: Determine that the first element to be matched and the second element to be matched are an element pair, and execute step 903.

[0283] When it is determined based on the position information that the first element to be matched matches the second element to be matched, the control component may determine that the first element to be matched and the second element to be matched are an element pair, and then may execute step 903 to match the next first element. Of course, if the first element to be matched is the last first element in the first element group, step 915 may be executed.

[0284] Step 915: After traversing the first element in the first element group, determine the monitoring data in the monitoring image based on at least one element pair.

[0285] After traversing the first element in the first element group, the control component can determine the monitoring data in the monitoring image based on at least one element pair. In the embodiment of the present application, after step 905 or step 914, if the currently selected first element is the last first element in the first element group, it indicates that the first elements in the first element group have been traversed, and this step can be performed.

[0286] In addition, this step can also refer to step 815 in the above embodiment, and the embodiment of the present application will not be repeated here.

[0287] In summary, the surveillance image processing method provided in the embodiments of the present application obtains a surveillance image at a first moment and, based on the surveillance image, obtains a first element group and a second element group. These element groups include position information of each element, and the first element and the second element are two elements of the person in the surveillance image. The first element and the second element can then be matched based on the position information to obtain multiple element pairs, and the surveillance data in the surveillance image can be confirmed based on the number of element pairs. In this way, surveillance data can be obtained based on the element pairs of the two elements, which can improve the accuracy of the obtained surveillance data, thereby achieving the effect of improving the accuracy of the surveillance image processing method.

[0288] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0289] FIG11 is a block diagram of a monitoring device provided in an embodiment of the present application. The monitoring device 1100 includes:

[0290] The image acquisition module 1110 is configured to acquire a monitoring image of the monitoring area at a first moment.

[0291] The element group acquisition module 1120 is used to obtain a first element group and a second element group in the surveillance image based on the surveillance image. The first element group includes the position information of at least one first element, and the second element group includes the position information of at least one second element. The first element and the second element are two elements of the person in the surveillance image.

[0292] The pairing module 1130 is configured to pair the first element and the second element based on the position information of the first element and the position information of the second element to obtain at least one element pair, where the element pair includes a first element and a second element that are associated with each other.

[0293] The data determination module 1140 is configured to determine the monitoring data in the monitoring image based on at least one element pair.

[0294] In summary, the monitoring device provided in the embodiments of the present application obtains a monitoring image at a first moment and, based on the monitoring image, obtains a first element group and a second element group. These element groups include position information of each element, and the first element and the second element are two elements of the person in the monitoring image. The first element and the second element can then be matched based on the position information to obtain multiple element pairs, and the monitoring data in the monitoring image can be confirmed based on the number of element pairs. In this way, monitoring data can be obtained based on the element pairs of the two elements, which can improve the accuracy of the obtained monitoring data, thereby achieving the effect of improving the accuracy of the monitoring image processing method.

[0295] In addition, an embodiment of the present application also provides a monitoring image processing device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the monitoring image processing method as described above.

[0296] An embodiment of the present application also provides a computer storage medium, in which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the monitoring image processing method as described above.

[0297] In addition, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the surveillance image processing methods provided in the various optional implementations described above.

[0298] In this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

[0299] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the facial attribute information involved in this application was obtained with full authorization.

[0300] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0301] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0302] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0303] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A monitoring image processing method, characterized in that: The method comprises: Acquire a monitoring image of the monitoring area at the first moment; Acquire a first element group and a second element group in the surveillance image, where the first element group includes position information of at least one first element in the surveillance image, and the second element group includes position information of at least one second element in the surveillance image, the first element and the second element being two elements of a person in the surveillance image; Pairing the first element and the second element based on the position information of the first element and the position information of the second element to obtain at least one element pair, where the element pair includes a first element and a second element that are associated with each other; Based on the at least one element pair, monitoring data in the monitoring image is determined.

2. The method according to claim 1, characterized in that The first element group includes position information of a plurality of first elements in the monitoring image, and the second element group includes position information of a plurality of second elements in the monitoring image; The pairing the first element and the second element based on the position information of the first element and the position information of the second element to obtain at least one element pair includes: Selecting a first element to be matched from the multiple first elements; Determine whether there is an associated second element with the first element to be matched; When there is no associated second element for the first element to be matched, selecting a second element to be matched from the multiple second elements; Determining whether the first element to be matched matches the second element to be matched based on the position information; During matching, determining that the first element to be matched and the second element to be matched are an element pair, and executing the step of selecting a first element to be matched from the plurality of first elements; When there is no match, the step of selecting a second element to be matched from the multiple second elements is performed.

3. The method according to claim 2, characterized in that The method further comprises: When the first element has an associated second element, determining whether the first element to be matched matches the associated second element based on the position information; During matching, determining that the first element to be matched and the associated second element are still an element pair; When there is no match, it is determined that the first element to be matched and the associated second element are not an element pair, and based on the position information, the associated second element is paired with other first elements in the first element group except the paired first element.

4. The method according to claim 2, characterized in that The second element is a human face. After obtaining the first element group and the second element group in the surveillance image based on the surveillance image, the method further includes: Acquire a tracking list and a lost tracking list based on the second element group, the tracking list including data of faces appearing in surveillance images at the first moment, and the lost tracking list including data of faces that appeared in surveillance images at a historical moment and did not appear in surveillance images during a preset time period starting from the historical moment, where the historical moment is a moment before the first moment; The method further comprises: When the first element has an associated second element, determining whether the associated second element is located in the tracking list; When located in the tracking list, determining whether the first element to be matched matches the associated second element based on the position information; During matching, determining that the first element to be matched and the associated second element are still an element pair; When there is no match, determining that the first element to be matched and the associated second element are not an element pair, and pairing the associated second element with other first elements in the first element group except the paired first element based on the position information; determining whether the associated second element is in the lost tracking list when it is not in the tracking list; When the second element is located in the lost tracking list, performing the step of selecting a second element to be matched from the plurality of second elements; When the first element to be matched and the associated second element are not in the lost tracking list, it is determined that the first element to be matched and the associated second element are still an element pair.

5. The method according to claim 2, characterized in that The second element is a human face, and before determining whether the first element to be matched matches the second element to be matched based on the position information, the method further includes: Determining whether the face quality of the image to be matched with the second element is greater than a preset face quality threshold; When the face quality is greater than the preset face quality threshold, the step of determining whether the first element to be matched matches the second element to be matched based on the position information is performed.

6. The method according to claim 2, characterized in that The determining, based on the position information, whether the first element to be matched matches the second element to be matched includes: Obtaining an overlapping area between an area occupied by the first element to be matched and an area occupied by the second element to be matched in the surveillance image; Whether the first element to be matched matches the second element to be matched is determined based on the size of the overlapping area.

7. The method according to claim 6, characterized in that The first element is a human head, the second element is a human face, and determining whether the first element to be matched matches the second element to be matched based on the size of the overlapping area includes: Determining an area ratio of the overlapping area to an area occupied by the first element to be matched in the monitoring image; When the area ratio is greater than or equal to a preset ratio, determining that the first element to be matched matches the second element to be matched; When the area ratio is smaller than the preset ratio, it is determined that the first element to be matched does not match the second element to be matched.

8. The method according to claim 2, characterized in that The selecting a first element to be matched from the multiple first elements includes: Selecting, from the multiple first elements, a first element to be matched that occupies the largest area in the surveillance image and has not been selected in the current matching, wherein the current matching is a matching between the first element and the second element based on the surveillance image at the first moment; When there is no associated second element for the first element to be matched, selecting a second element to be matched from the plurality of second elements includes: When there is no associated second element with the first element to be matched, the second element to be matched that occupies the largest area in the monitoring image and has not been selected in this matching is selected from the multiple second elements.

9. The method according to claim 1, characterized in that The acquiring, based on the surveillance image, a first element group and a second element group in the surveillance image includes: Acquire images of a plurality of first elements and images of a plurality of second elements in the surveillance image; Arranging the plurality of first elements from largest to smallest in accordance with their sizes in the monitoring image to obtain a first element column; Obtaining the first element group based on position information of first n first elements in the first element column, where n is an integer greater than zero; Arranging the plurality of second elements in descending order of size to obtain a second element column; The second element group is obtained based on position information of the first n second elements in the second element column.

10. The method according to claim 9, characterized in that The second element is a human face. Before arranging the plurality of second elements in descending order of size to obtain a second element column, the method further includes: The second elements whose image has a face quality less than a preset face quality threshold are removed from the multiple second elements.

11. The method according to any one of claims 1 to 10, characterized in that: The second element is a human face, and determining the monitoring data in the monitoring image based on the at least one element pair includes: Based on the image of the second element in the element pair, facial attribute information of the second element is obtained.

12. The method according to any one of claims 1 to 10, characterized in that: Determining monitoring data in the monitoring image based on the at least one element pair includes: The number of people in the surveillance image is determined according to the number of the at least one element pair.

13. A monitoring device, characterized in that: The monitoring device comprises: An image acquisition module, configured to acquire a monitoring image of the monitoring area at a first moment; an element group acquisition module, configured to acquire, based on the surveillance image, a first element group and a second element group in the surveillance image, wherein the first element group includes position information of at least one first element, the second element group includes position information of at least one second element, and the first element and the second element are two elements of a person in the surveillance image; a pairing module, configured to pair the first element and the second element based on the position information of the first element and the position information of the second element, to obtain at least one element pair, the element pair including a first element and a second element that are associated with each other; A data determination module is used to determine the monitoring data in the monitoring image based on the at least one element pair.

14. A monitoring device, characterized in that: The monitoring image processing device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the monitoring image processing method according to any one of claims 1 to 12.

15. A non-volatile computer storage medium, characterized in that: The computer storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the monitoring image processing method as described in any one of claims 1 to 12.

Images