A method, apparatus and device for determining a sensitive area of driving scene data

Abstract

The embodiment of the specification discloses a method, device and equipment for determining a sensitive area of driving scene data, and relates to the technical field of image processing. The method comprises the following steps: detecting a target object by using a global detection model, the target object comprising a first object and a second object, the global detection model comprising a backbone network, a first detection head and a second detection head, the backbone network being used for feature extraction on picture data; the first detection head is a binary classification detection head used for detecting the first object, and the second detection head is a binary classification detection head used for detecting the second object; performing feature extraction on the obtained picture data by using the backbone network to obtain image features; detecting the first object and the second object based on the image features by using the first detection head and the second detection head respectively, and determining an area occupied by the first object and the second object as a first recognition area; and the first recognition area is a sensitive area of the picture data. The sensitive area can be quickly and accurately determined.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and device for determining sensitive areas of driving scene data. Background Technology

[0002] With the increasing development of autonomous driving technology, the requirements for data compliance and data security in the development of environmental perception and related product applications of intelligent connected vehicles are also becoming higher and higher. In many cases, the collected driving scene data needs to be de-identified before it can be used.

[0003] When desensitizing images, the areas occupied by faces or license plates are first identified and designated as sensitive areas. These sensitive areas are then desensitized. Similarly, when desensitizing videos, each frame of the video is desensitized. Accurately and quickly identifying sensitive areas is crucial for removing sensitive information to the greatest extent possible without damaging other areas, thus ensuring the subsequent use of the image data.

[0004] In existing technologies, two detection models are used to identify sensitive regions: one model includes a feature extractor and a face detection head, and the other includes a feature extractor and a license plate detection head. When detecting image data, one detection model first extracts features from the image to obtain first image features, and then a face detection head detects these first image features to identify face regions. Then, the other detection model extracts features from the image to obtain second image features, and then a license plate detection head detects these second image features to identify license plate regions. The feature extraction process for images is computationally intensive and time-consuming. Furthermore, existing technologies require two feature extraction steps, resulting in high computational requirements and slow processing speeds.

[0005] Therefore, there is an urgent need for a method to determine the sensitive areas of driving scene data, which can quickly and accurately identify the sensitive areas. Summary of the Invention

[0006] To address the aforementioned technical problems, embodiments of this specification propose a method, apparatus, and device for determining sensitive areas in driving scene data, which can quickly and accurately identify sensitive areas.

[0007] This specification provides a method for determining sensitive areas of driving scene data. It employs a global detection model to detect target objects, including a first object and a second object. The global detection model includes a backbone network, a first detection head, and a second detection head. The backbone network is used for feature extraction from image data. The first detection head is a binary classification detection head used to detect the first object, and the second detection head is a binary classification detection head used to detect the second object. The method includes:

[0008] The backbone network is used to extract features from the acquired image data to obtain image features;

[0009] The first detection head and the second detection head are used respectively to detect the first object and the second object based on the image features, and the area occupied by the first object and the second object is determined as the first recognition area; the first recognition area is the sensitive area of ​​the image data.

[0010] This specification provides a computer device that uses a global detection model to detect target objects. The target objects include a first object and a second object. The global detection model includes a backbone network, a first detection head, and a second detection head. The backbone network is used for feature extraction from image data. The first detection head is a binary classification detection head used to detect the first object, and the second detection head is a binary classification detection head used to detect the second object. The device includes:

[0011] The feature extraction module is used to extract features from the acquired image data using the backbone network to obtain image features;

[0012] The first detection module is used to detect the first object and the second object based on the image features using the first detection head and the second detection head respectively, and to determine the area occupied by the first object and the second object as the first recognition area; the first recognition area is the sensitive area of ​​the image data.

[0013] This specification provides an embodiment of a computer device / equipment / system, including a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the steps of any of the methods described above.

[0014] This specification provides an embodiment of a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0015] This specification provides an embodiment of a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of any of the methods described above.

[0016] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0017] This specification discloses a method, apparatus, and device for determining sensitive regions in driving scene data. When using a global detection model to detect target objects, only one feature extraction is needed on the image data to obtain image features. A first detection head and a second detection head then detect a first object and a second object based on these image features, thereby determining a first recognition region. The first recognition region is the sensitive region of the image data. Since feature extraction is the most computationally intensive and time-consuming part of the entire sensitive region determination process, this method, compared to existing technologies, reduces the number of feature extractions, avoids redundant calculations, has a lower computational load, and a faster processing speed, enabling rapid and accurate identification of face regions and license plate regions. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for determining sensitive areas of driving scene data provided in an embodiment of this specification;

[0020] Figure 2 This is a schematic diagram of the structure of a global detection model provided in an embodiment of this specification;

[0021] Figure 3 This is a schematic diagram of the structure of a second local detection model provided in the embodiments of this specification;

[0022] Figure 4 This is a schematic diagram illustrating the detection of consecutive video frame images in video data, provided by an embodiment of this description.

[0023] Figure 5 This illustration shows a method for detecting non-sampled frame images in video data, as provided in this embodiment.

[0024] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this specification;

[0025] Figure 7 This is a schematic diagram of the structure of a computer device / equipment / system provided for an embodiment of this specification. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0027] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this invention, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0028] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

[0029] To address the shortcomings of existing technologies, this solution provides the following embodiments:

[0030] Figure 1 This is a flowchart illustrating a method for determining sensitive areas of driving scene data, provided in an embodiment of this specification.

[0031] From a hardware perspective, the entity executing this process can be a device that identifies sensitive areas of driving scenario data. From a programmatic perspective, it can be an application running on the device or an SDK (Software Development Kit) deployed on a server.

[0032] like Figure 1 As shown, the process may include the following steps:

[0033] Step 101: Use the backbone network to extract features from the acquired image data to obtain image features.

[0034] Image data can be single images, which are then used to identify sensitive areas. Image data can also be video frame images from video data. When desensitizing video data, video frame images can be obtained frame by frame, and sensitive areas can be identified for each frame.

[0035] Input the image data into the global detection model. Figure 2 This is a schematic diagram of the structure of a global detection model provided in an embodiment of this specification. Figure 2 As shown, the global detection model includes a backbone network 1, a first detection head 2, and a second detection head 3. After image data is input into the global detection model, the backbone network extracts features from the image data 4 to obtain image features. These image features are then fed into the first and second detection heads respectively. The backbone network can be a ResNet network or a VGGNet network, among other network structures.

[0036] Step 103: The first detection head and the second detection head are used respectively to detect the first object and the second object based on the image features, and the area occupied by the first object and the second object is determined as the first recognition area; the first recognition area is the sensitive area of ​​the image data.

[0037] The first object can be a face. The first detection head is a binary classification detection head, used to distinguish between empty spaces and faces, with empty spaces representing non-faces. The first detection head performs detection based on image features extracted by the backbone network; after detecting a face, it can generate detection boxes surrounding the face, and the area surrounded by the detection boxes is the area occupied by the face, which is the first recognition area.

[0038] The second object can be a license plate. The second detection head is a binary classification detection head used to distinguish between empty spaces and license plates; empty spaces represent non-license plates. The second detection head performs detection based on image features extracted from the backbone network; after detecting the license plate, it can generate a series of detection boxes surrounding the license plate. The area surrounded by the detection boxes is the area occupied by the license plate, which is the first recognition area.

[0039] Figure 1 The proposed method employs a global detection model to detect target objects. It only requires one feature extraction operation on the image data to obtain image features. A first detection head and a second detection head then use these image features to detect the first and second objects, respectively, thereby determining the first recognition region, which is the sensitive region. Since feature extraction is the most computationally intensive and time-consuming part of the entire sensitive region determination process, this method, compared to existing technologies, reduces the number of feature extraction operations, avoids redundant calculations, has a lower computational load, and a faster processing speed, enabling rapid and accurate identification of face and license plate regions.

[0040] It should be noted that, Figure 1 The sensitive areas of image data identified by the method described can be used in image data anonymization applications. When anonymizing image data, simply add a step to anonymize the identified sensitive areas. However, this should not be interpreted as a limitation on the use of the identified sensitive areas of image data; they can also be used for other purposes.

[0041] Optionally, the image data is video frame image data from video data; the method further includes:

[0042] Based on motion estimation of target objects detected from video frame image data preceding the video frame image data, a preliminary missed region is determined where the global detection model missed detection during the video frame image data detection process; the preliminary missed region represents the area occupied by the undetected target object;

[0043] The preliminary omission area was detected and verified using a local detection model;

[0044] If a target object exists in the initially missed area, the area occupied by the target object is determined as the second identification area; the second identification area is a sensitive area of ​​the video frame image data.

[0045] An image dataset may contain multiple target objects, and a global detection model is used to detect all of them. However, the global detection model may miss some targets. To further improve the recall rate of this method, i.e., to minimize the number of missed targets, after the global detection model completes its detection, preliminary missed regions can be identified. Specifically, when the image data is video frame data (i.e., when determining sensitive regions frame by frame), since there are connections between consecutive video frames, preliminary missed regions can be identified by motion estimation based on previously detected target objects in the current video frame. Then, a local detection model is used to verify the preliminary missed regions, checking whether target objects exist within them. Missed objects are detected, de-identified, and their recall rate is improved, while the false negative rate is reduced.

[0046] Local detection models only target initially missed regions, and these local regions are detected. Compared to global detection models, they require less computation and are faster. This method improves recall and reduces the omission of target objects while minimizing time and computation.

[0047] Optionally, determining the preliminary missed regions that the global detection model missed during the detection process specifically includes:

[0048] A first prediction region is determined from the video frame image data. The first prediction region is the region occupied by the first tracking object in the video frame image data, which is obtained by motion estimation and prediction of the first tracking object. The first tracking object is a target object detected from a first preset number of video frame image data preceding the video frame image data.

[0049] The first predicted region is matched with the first identified region to obtain a first predicted region that does not match the first identified region. The first predicted region that does not match the first identified region is determined as the preliminary missed region.

[0050] When the image data is video frame image data from video data, meaning during the anonymization of video data, the target objects in each video frame image have logical connections. For example, a target object in a previous video frame image may appear in the next video frame image, only in a different position. Thus, tracking logic can be added between video frames. When detecting the current video frame image, the target object detected from the previous video frames image can be identified as the first tracking object. Based on the position information of the first tracking object in each video frame image, its motion speed, and motion direction, the position of the first tracking object in the current video frame image is estimated. A predicted detection box is generated at the estimated position, and the area enclosed by the predicted detection box is the first predicted region. The first predicted region is fused and matched with the first recognized region. If there is a first predicted region that does not match the first recognized region, it may be because the first tracking object corresponding to this first predicted region was missed by the global detection model, or it may be because the motion estimation of the first tracking object is not accurate enough. This first predicted region is identified as a preliminary missed region for further detection and verification by the local detection model.

[0051] The method for fusing and matching the first predicted region and the first identified region can specifically be to determine the position scores of the two regions. When the difference between the position scores of the two regions is greater than a preset threshold, it indicates that the match has not been successful. When the difference between the position scores of the two regions is less than the preset threshold, it indicates that the match has been successful.

[0052] The first preset quantity can be set as needed; for example, the first preset quantity can be 1, 2, or more.

[0053] Optionally, the step of using a local detection model to detect and verify the initially missed area specifically includes:

[0054] Obtain first partial image data; the first partial image data is the partial image data of the video frame image data corresponding to the region after the initial omission region is expanded;

[0055] A local detection model is used to detect target objects in the local image data.

[0056] Optionally, the local detection model includes a first local detection model and a second local detection model, wherein the first local detection model is a single-target detection model for detecting the first object, and the second local detection model is a single-target detection model for detecting the second object;

[0057] The step of using a local detection model to detect target objects in the local image data specifically includes:

[0058] When the target object corresponding to the first local image data is the first object, the first local detection model is used to detect the target object in the first local image data.

[0059] When the target object corresponding to the first local image data is the second object, the second local detection model is used to detect the target object in the first local image data.

[0060] There can be two local detection models: one for detecting the first object and the other for detecting the second object. Figure 3 This is a schematic diagram of the structure of a second local detection model provided in an embodiment of this specification. Figure 3 As shown, the second local detection model includes a second local backbone network 5 and a second local detection head 6. The second local backbone network is used to extract image features of the second object from the first local image data 7. The first local image data 7 can be a local image data cropped from image data 4. The second local detection head is a binary classification detection head used to detect the second object. It should be noted that the second local backbone network can use a relatively simple ResNet network, and the complexity of the network structure is much less than that of the backbone network of the global detection model. The first local detection model is similar to the second local detection model, including a first local backbone network and a first local detection head; however, the first local backbone network of the first local detection model is used to extract image features of the first object, and the first local detection head is used to detect the first object. That is, the local detection model is a single-object detection model, which can be a binary classifier regressor based on a convolutional neural network, used to detect local regions of image data, with low computational cost, fast detection speed, and high judgment accuracy.

[0061] When using the motion estimation and prediction methods for the target object mentioned above to determine the initial missed area, motion estimation and prediction are performed on each first object and each second object to obtain a predicted detection box. The predicted detection boxes that do not match the detection boxes are appropriately enlarged, and the image data of the enlarged area is cropped out as the first local image data. Then, it is input into the corresponding detection model for detection and verification to check whether the first local image data contains the target object.

[0062] Specifically, refer to Figure 4 The methods for determining the initial omission area and for detecting and verifying the initial omission area are illustrated with examples. Figure 4 This illustration shows a method for detecting consecutive video frame images in video data, as provided in this embodiment. Figure 4 As shown, the target objects detected in the first video frame are A1B1C1D1, where A1B1 is the first object and C1D1 is the second object. The first recognition region is represented by a solid bounding box surrounding each target object. When detecting in the second video frame, A1B1C1D1 is identified as the first tracking object. Motion estimation is performed on the first tracking object to predict the region it occupies in the second video frame, which is the first prediction region, A2B2C2D2, represented by a dashed bounding box. It should be noted that the dashed bounding box and the solid bounding box can have the same size. Figure 4 For ease of description, the dashed bounding box is drawn larger than the solid bounding box. The second video frame is detected using a global detection model, and the detected target object is A1B1C1E1. The area occupied by A1B1C1E1 is defined as the first recognition region, which is a sensitive region. At this point, D1 actually still exists in the second video frame, but it has not been detected by the global detection model (the solid bounding box of D1 exists, but it is not marked to indicate that it has not been detected by the global detection model). The first predicted region and the first recognition region are fused and matched. A2 matches A1, B2 matches B1, and C2 matches C1 successfully. D2 fails to match the first recognition region and is identified as a preliminary missed region. D2 is then enlarged to D3. The enlargement ratio can be a first ratio, which can be determined according to actual needs. The image data at D3 is cropped as the first local image data. Since D1 is the second object, D3 is input into the second local detection model for detection and verification. The second local detection model will detect D1 and determine the area occupied by D1 as the second recognition area, which is also the sensitive area.

[0063] When detecting the third video frame, the first preset quantity is set to 1 when determining the first tracking object. Therefore, only the target object A1B1C1D1E1 detected in the second frame is identified as the first tracking object. The target object detected by the global detection model is also A1B1D1E1. It should be noted that C1 does not appear in the third video frame and is not detected by the global detection model. After fusion matching, C2 fails to match and is identified as a preliminary missed area. C2 is expanded to obtain C3, and the image data at C3 is cropped as the first local image data. Since C1 is the second object, C3 is input into the second local detection model for detection and verification. The absence of the target object indicates an error in the motion estimation and prediction of C1. C1 may have left the video capture area and will not appear in subsequent video frames.

[0064] When detecting the fourth video frame, if the first preset number is set to 1, then only the target object A1B1D1E1 detected from the third video frame will be identified as the first tracking object. If the first preset number is set to 2, then the target object A1B1C1D1E1 detected from both the second and third video frames will be identified as the first tracking object. One specific implementation method is as follows: After detecting a target object, assign an ID to each target object. For example, after detecting a target object in the first video frame, assign an ID A1B1C1D1 to each target object, creating an ID set. Add A1B1C1D1 to the ID set. Target objects in the ID set are identified as the first tracking objects. Then, detect the second video frame. The detected target object is A1B1C1D1E1. A1B1C1D1 matches A2B2C2D2 successfully, indicating that A1B1C1D1 is a previously detected target object, and its original ID can be used. E1 does not match the first predicted region, indicating that E1 has not appeared before and is a new target object. Assign a new ID, E1, to it. Add E1 to the ID set. A1B1C1D1E1 is identified as the first tracking object. When detecting the third video frame, if the local detection model fails to detect the target object in C3, and the first preset quantity is 1, then C1 is removed from the ID set. When detecting the fourth video frame, C1 is no longer considered the first tracking object. If the second preset quantity is 2 (two objects), then C1 is not directly removed from the ID set and continues to be considered the first tracking object. If the target object is still not detected in C3 in the next video frame (meaning C1 does not appear in two consecutive detections), then C1 is removed from the ID set.

[0065] Optionally, the image data is sampled frame image data obtained by intermittently sampling the video frame image data; the sampling frequency of the intermittent sampling is determined based on the number of the first object and the second object, as well as the refresh rate of the video data.

[0066] The sampling frequency can be determined based on the number of the first object and the second object, and the refresh rate of the video data. In other words, it is determined based on the complexity of the driving scene data and the refresh rate of the video data. When the driving scene data is more complex, the sampling frequency can be increased. When the driving scene data is simpler, the sampling frequency can be decreased. Specifically, formula (1) can be used to determine the video sampling frequency under different scene complexities.

[0067] s = P k (1)

[0068] Because the video frames need to be evenly divided, it is necessary to ensure that the sampling frequency is divisible by the refresh rate (fps). Therefore, the actual sampling frequency should be a factor of fps, where P is the list of factors of fps, and the subscript k is determined by formula (2).

[0069]

[0070] For example, fps = 24, P(fps) = [1,2,3,4,6,8,12]. Based on previous detection results, the current scene has 1 face target and 3 license plate targets, so k = 4, therefore s = P4 = 6 (k starts counting from 0). That is, 6 frames are sampled per second, and one frame is sampled every 4 frames. m is the number of faces, and n is the number of license plates. Formula (2) defines a scene with more than 5 faces and more than 8 license plates in the video data as a complex scene.

[0071] Optionally, the method further includes determining sensitive regions of non-sampled frame image data; the non-sampled frame image data refers to video frame image data other than the sampled frame image data in the video frame image data; determining the sensitive regions of non-sampled frame image data specifically includes:

[0072] Acquire non-sampled frame image data;

[0073] A second prediction region is determined from the unsampled frame image data. The second prediction region is the region occupied by the second tracking object in the unsampled frame image data, which is obtained by motion estimation and prediction of the second tracking object. The second tracking object is a target object detected from a second preset number of video frame image data preceding the unsampled frame image data.

[0074] Obtain the second local image data; the second local image data is the local image data of the non-sampled frame image data corresponding to the region after the second prediction region is expanded;

[0075] The local detection model is used to detect the target object in the second local image data, and the area occupied by the target object is determined as the third recognition area; the third recognition area is the sensitive area of ​​the non-sampled frame image data.

[0076] When desensitizing video data, each video frame can be used as a sample frame, meaning each video frame is sampled. Sensitive regions are then determined for each video frame using any of the methods mentioned above. Alternatively, the video data can be sampled at intervals, with one frame sampled every few frames. Sensitive regions are then determined for the sampled frames using any of the methods mentioned above. Then, other methods are used to determine the sensitive regions for the non-sampled frame data. The non-sampled frame data consists of video frames other than the sampled frames.

[0077] The method for determining sensitive regions in non-sampled frame images can avoid using a global detection model to detect target objects. Instead, it directly identifies target objects detected previously in the non-sampled frame image as the second tracking object. Motion estimation and prediction of the second tracking object's position in the current non-sampled frame image yield a second predicted region. Each second predicted region is appropriately enlarged, and the enlarged region is cropped from the non-sampled frame image as second local image data, which is then input into the corresponding local detection model for detection. If the second predicted region is predicted to be the first object, the second local image data is fed into the first local detection model; if it is predicted to be the second object, it is fed into the second local detection model. After detection, a third recognition region is obtained, which is the sensitive region of the non-sampled frame image.

[0078] Reference Figure 5 To illustrate, Figure 5 This illustration demonstrates a method for detecting non-sampled frame images in video data, as provided in this embodiment. The first and fourth frames are sampled frame images, while the second and third frames are non-sampled frame images. When detecting the second frame, motion estimation is directly performed on the target object F1G1H1 detected from the first frame to obtain a second prediction region F2G2H2. This second prediction region is then expanded to obtain second local image data F3G3H3, which are input into the corresponding local detection models for detection, resulting in a third recognition region, which is the sensitive region.

[0079] Therefore, when desensitizing video data, only the sampled frame images in the video data need to be detected using a global detection model, while the non-sampled frame images need to be detected using a local detection model. This reduces the use of the global detection model, since the computational load and detection time of the global detection model are greater than those of the local detection model. This reduces the computational load and improves the efficiency of identifying sensitive areas.

[0080] Optionally, the method further includes:

[0081] Determine whether a new target object appears in the first sampled frame image data after the non-sampled frame image data, and obtain the first determination result;

[0082] If the first judgment result indicates that a new target object appears in the first sampled frame image data after the non-sampled frame image data, then the global detection model is used to detect the target object in the non-sampled frame image, and the area occupied by the target object is determined as the fourth recognition area; the fourth recognition area is the sensitive area of ​​the non-sampled frame image data.

[0083] When determining sensitive regions for unsampled frame images, a situation may arise where a new target object is detected in the first sampled frame image following the unsampled frame image. This new target object may have already appeared in the unsampled frame image. However, since the second prediction region is obtained by tracking, motion estimation, and prediction of target objects detected in video frames preceding the unsampled frame image (which can be only sampled frames or include both sampled and unsampled frames), the previously detected target objects naturally do not include the newly appearing target object, thus failing to track it and resulting in a missed detection. In this case, the unsampled frame image can be backtracked for re-detection. During re-detection, a global detection model can be used to re-detect the unsampled frame image. The area occupied by the detected target object is defined as the fourth recognition region, which is also a sensitive region. Both the third and fourth recognition regions are sensitive regions.

[0084] Optionally, determining whether a new target object appears in the first sampled frame image data after the non-sampled frame image specifically includes:

[0085] Determine the first prediction region and the first recognition region of the sampled frame image data;

[0086] The first identification region is matched with the first prediction region to determine whether there is a first identification region that does not match the first prediction region.

[0087] If there is a first identification region that does not match the first predicted region, it indicates that a new target object has appeared.

[0088] The method for determining the first prediction region of the sampled frame image data is the method for determining the first prediction region described above.

[0089] Reference Figure 5 To clarify, the fourth video frame is the first sampled frame after the second and third video frames. The first identified region detected by the global detection model in the fourth video frame is F1G1H1I1, and the first predicted region is F2G2H2. F1G1H1 and F2G2H2 match successfully, but I1 fails to match the first predicted region, indicating that I1 is a new target object. After detecting a new target object, it can be inferred that this new target object I1 may have already appeared in the second or third video frame. However, since I1 did not appear in the first video frame, it could not be detected by the global detection model and therefore was not identified as the second tracking object, potentially resulting in a missed detection. Therefore, the second and third video frames need to be re-detected using the global detection model to identify I1 in the third video frame.

[0090] Optionally, the training method for the global detection model specifically includes:

[0091] The first detection model is trained using a multi-class labeled dataset to obtain a trained backbone network; the first detection model consists of a backbone network and a third detection head, wherein the third detection head is a multi-class detection head;

[0092] The parameters of the trained backbone network are fixed, and the second detection model is trained using the dataset labeled with the first object to obtain the trained first detection head; the second detection model consists of the trained backbone network and the first detection head.

[0093] The parameters of the trained backbone network are fixed, and the third detection model is trained using the dataset labeled with the second object to obtain the trained second detection head; the third detection model consists of the trained backbone network and the second detection head.

[0094] During training, the backbone network is trained first. The backbone network and the multi-class detection head are then combined to form the first detection model, which is trained using a multi-class labeled dataset. This multi-class labeled dataset may not contain faces or license plates, or may not include these labels. The data scenario of this multi-class labeled dataset only needs to be similar to a driving scenario; for example, labels may be added for vehicles, lane lines, etc. When the first detection model converges, meaning it can accurately identify the labeled objects in the dataset, it indicates that the backbone network's feature extraction function for image data has been successfully trained. This multi-class labeled dataset can be the BDD100K dataset.

[0095] After the backbone network is trained, its parameters are fixed. It is then combined with the first and second detection heads to form the second and third detection models, respectively.

[0096] The second detection model is trained using a dataset labeled with the first object until the second detection model converges, meaning it can accurately identify the labeled first object, indicating that the first detection head has completed training. This dataset with labeled first objects is a single-class labeled dataset; it can be used to label only the first object, and the second detection model only needs to distinguish between empty spaces and the first object, i.e., it only needs to determine whether it is the first object. Specifically, this dataset can be the Wideface dataset.

[0097] The third detection model is trained using a dataset labeled with the second object until the training converges, meaning the third detection model can accurately identify the labeled second object, indicating that the second detection head has completed training. This dataset with labeled second objects is a single-class labeled dataset; it only needs to label the second object, and the third detection model only needs to distinguish between empty spaces and the second object, i.e., it only needs to determine whether it is the second object. Specifically, this dataset can be the CCPC dataset.

[0098] The trained backbone network, the first detection head, and the second detection head are combined into a global detection model.

[0099] Because the parameters of the backbone network are fixed during the training of the second and third detection models, when using a global model to detect target objects in image data of driving scenes, the image features extracted by the backbone network from the image data can be applied to both the first and second detection heads. That is, the first detection head can detect the first target object based on these image features, and the second detection head can detect the second target object based on these image features.

[0100] One existing technology proposes a detection model that includes a feature extractor and a three-class detection head, capable of distinguishing between empty spaces, faces, and license plates. The model inputs image data, the feature extractor extracts image features, and the detection head identifies faces and license plates by performing a single detection on these features. However, this model requires training on a dataset that annotates both faces and license plates. Currently available public datasets do not contain such annotated datasets. Therefore, firstly, data needs to be collected and annotated again to train the detection model, resulting in high development and annotation costs. Secondly, the collected images of driving scenes will contain far more license plates than faces, leading to an imbalanced training sample and negatively impacting the model's performance.

[0101] The method of this invention can train the first detection model using existing publicly available multi-class labeled datasets. This training process primarily trains the feature extraction function of the backbone network, rather than enabling the first detection model to detect faces and license plates. Therefore, the multi-class labeled dataset may not contain faces and license plates, or may not be labeled at all. Thus, existing multi-class labeled datasets can be used for training. When training the second detection model, existing face-labeled datasets can be used, enabling the first detection head to accurately identify faces. When training the third detection model, existing license plate-labeled datasets can be used, enabling the second detection head to accurately identify license plates. In this way, firstly, existing datasets can be used to train the global detection model, saving development and labeling costs. Secondly, using different datasets for training the first and second detection heads ensures that the amount of training data for both is similar, thereby avoiding imbalanced samples that could affect the performance of the detection models.

[0102] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods.

[0103] Figure 6 This is a schematic diagram of the structure of a computer device provided as an embodiment of this specification. Figure 6 As shown, the device can use a global detection model to detect target objects, including a first object and a second object. The global detection model includes a backbone network, a first detection head, and a second detection head. The backbone network is used for feature extraction from image data. The first detection head is a binary classification detection head used to detect the first object, and the second detection head is a binary classification detection head used to detect the second object. It includes:

[0104] The feature extraction module 601 is used to extract features from the acquired image data using the backbone network to obtain image features.

[0105] First detection module 602: used to detect the first object and the second object based on the image features using the first detection head and the second detection head respectively, and to determine the area occupied by the first object and the second object as the first recognition area; the first recognition area is the sensitive area of ​​the image data.

[0106] Optionally, the device further includes:

[0107] The second determining module is used to determine, based on motion estimation of target objects detected from video frame image data preceding the video frame image data, a preliminary missed region where the global detection model missed detection during the detection of the video frame image data; the preliminary missed region represents the area occupied by the undetected target object;

[0108] The second detection module is used to detect and verify the preliminary missed area using a local detection model;

[0109] If a target object exists in the initially missed area, the area occupied by the target object is determined as the second identification area; the second identification area is the sensitive area of ​​the video frame image data.

[0110] Optionally, the second determining module specifically includes a third determining module and a first matching module;

[0111] The third determining module is used to determine a first prediction region of the video frame image data. The first prediction region is the region occupied by the first tracking object in the video frame image data, obtained by motion estimation and prediction of the first tracking object. The first tracking object is a target object detected from a first preset number of video frame image data preceding the video frame image data.

[0112] The first matching module is used to match the first predicted region with the first identified region to obtain a first predicted region that does not match the first identified region, and to determine the first predicted region that does not match the first identified region as the preliminary missed region.

[0113] Optionally, the second detection module includes a first acquisition module and a third detection module;

[0114] The first acquisition module is used to acquire first partial image data; the first partial image data is partial image data of the video frame image data corresponding to the region after the initial omission region is expanded;

[0115] The third detection module is used to perform target object detection on the local image data using a local detection model.

[0116] Optionally, the third detection module includes a fourth detection module and a fifth detection module;

[0117] The fourth detection module is used to perform target object detection on the first local image data using the first local detection model when the target object corresponding to the first local image data is the first object.

[0118] The fifth detection module is used to perform target object detection on the first local image data using a second local detection model when the target object corresponding to the first local image data is the second object.

[0119] Optionally, the device further includes:

[0120] The second acquisition module is used to acquire non-sampled frame image data;

[0121] The fourth determining module is used to determine the second prediction region of the non-sampled frame image data. The second prediction region is the region occupied by the second tracking object in the video frame image data, which is obtained by motion estimation and prediction of the second tracking object. The second tracking object is a target object detected from a second preset number of video frame image data preceding the non-sampled frame image data.

[0122] The third acquisition module is used to acquire second local image data; the second local image data is the local image data of the non-sampled frame image data corresponding to the region after the second prediction region is expanded;

[0123] The sixth detection module is used to detect the target object in the second local image data using the local detection model, and to determine the area occupied by the target object as the third recognition area; the third recognition area is the sensitive area of ​​the non-sampled frame image data.

[0124] Optionally, the device further includes:

[0125] The first judgment module is used to determine whether a new target object appears in the first sampled frame image data after the non-sampled frame image data, and to obtain a first judgment result.

[0126] The seventh detection module is used to detect the target object in the non-sampled frame image if the first judgment result indicates that a new target object appears in the first sampled frame image data after the non-sampled frame image data, and to determine the area occupied by the target object as the fourth recognition area; the fourth recognition area is the sensitive area of ​​the non-sampled frame image data.

[0127] Optionally, the first judgment module is specifically used to match the first identification region with the first prediction region to determine whether there is a first identification region that has not been successfully matched with the first prediction region.

[0128] Based on the same line of thought, referring to Figure 7 This specification also provides a computer device / equipment / system 700, including a memory 730, a processor 710, and a computer program / instructions 720 stored in the memory, wherein the processor executes the computer program / instructions to implement the steps of any of the methods described above.

[0129] Based on the same idea, embodiments of this specification also provide a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0130] Based on the same idea, embodiments of this specification also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0131] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. Related parts can be referenced interchangeably.

[0132] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0133] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the memory's control logic. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means that can be included within it to implement various functions can also be considered as structures within the hardware component. Alternatively, the means that can be used to implement various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0134] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0135] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0136] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (which may include, but are not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine that can be used to implement the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture that may include instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that can be used to implement a process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] In a typical configuration, a computing device may include one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0142] Computer-readable media can include both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media can include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital character versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media cannot include transient computer-readable media, such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "may include," "comprise," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that may include a list of elements may include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "may include a…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that may include said element.

[0144] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (which may include, but are not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules can include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0146] The above description is merely an embodiment of this application and should not be construed as limiting the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for determining sensitive areas of driving scene data, characterized in that, A global detection model is used to detect target objects, which include a first object and a second object. The global detection model includes a backbone network, a first detection head, and a second detection head. The backbone network is used to extract features from image data. The first detection head is a binary classification detection head for detecting the first object, and the second detection head is a binary classification detection head for detecting the second object; including: The backbone network is used to extract features from the acquired image data to obtain image features; The first detection head and the second detection head are used respectively to detect the first object and the second object based on the image features, and the areas occupied by the first object and the second object are determined as the first recognition area; the first recognition area is the sensitive area of ​​the image data; wherein, the first object is a face; the second object is a license plate; The image data is video frame image data from video data; based on motion estimation of target objects detected from video frame image data preceding the video frame image data, a preliminary missed area is determined where the global detection model missed detection during the detection process; a local detection model is used to detect and verify whether a target object exists in the preliminary missed area to determine a second identification area; the preliminary missed area represents the area occupied by the undetected target object; the second identification area is a sensitive area of ​​the video frame image data.

2. The method as described in claim 1, characterized in that, The step of using a local detection model to detect and verify whether a target object exists in the initially missed region in order to determine the second identification region includes: The preliminary omission area was detected and verified using a local detection model; If a target object exists in the initially missed area, the area occupied by the target object is determined as the second identification area.

3. The method as described in claim 2, characterized in that, The determination of the preliminary omission region where the global detection model has missed detection during the detection process specifically includes: A first prediction region is determined from the video frame image data. The first prediction region is the region occupied by the first tracking object in the video frame image data, which is obtained by motion estimation and prediction of the first tracking object. The first tracking object is a target object detected from a first preset number of video frame image data preceding the video frame image data. The first predicted region is matched with the first identified region to obtain a first predicted region that does not match the first identified region. The first predicted region that does not match the first identified region is determined as the preliminary missed region.

4. The method as described in claim 2, characterized in that, The step of using a local detection model to detect and verify the initially missed areas specifically includes: Obtain first partial image data; the first partial image data is the partial image data of the video frame image data corresponding to the region after the initial omission region is expanded; A local detection model is used to detect target objects in the local image data.

5. The method as described in claim 4, characterized in that, The local detection model includes a first local detection model and a second local detection model. The first local detection model is a single-target detection model for detecting the first object, and the second local detection model is a single-target detection model for detecting the second object. The step of using a local detection model to detect target objects in the local image data specifically includes: When the target object corresponding to the first local image data is the first object, the first local detection model is used to detect the target object in the first local image data. When the target object corresponding to the first local image data is the second object, the second local detection model is used to detect the target object in the first local image data.

6. The method as described in claim 3, characterized in that, The image data is sampled frame image data obtained by intermittently sampling the video frame image data; the sampling frequency of the intermittent sampling is determined based on the number of the first object and the second object, as well as the refresh rate of the video data.

7. The method as described in claim 6, characterized in that, The method further includes determining the sensitive regions of non-sampled frame image data; the non-sampled frame image data refers to video frame image data other than the sampled frame image data in the video frame image data. The determination of sensitive regions in non-sampled frame image data specifically includes: Acquire non-sampled frame image data; A second prediction region is determined from the unsampled frame image data. The second prediction region is the region occupied by the second tracking object in the video frame image data, which is obtained by motion estimation and prediction of the second tracking object. The second tracking object is a target object detected from a second preset number of video frame image data preceding the unsampled frame image data. Obtain the second local image data; the second local image data is the local image data of the non-sampled frame image data corresponding to the region after the second prediction region is expanded; The local detection model is used to detect the target object in the second local image data, and the area occupied by the target object is determined as the third recognition area; the third recognition area is the sensitive area of ​​the non-sampled frame image data.

8. The method as described in claim 7, characterized in that, The method further includes: Determine whether a new target object appears in the first sampled frame image data after the non-sampled frame image data, and obtain the first determination result; If the first judgment result indicates that a new target object appears in the first sampled frame image data after the non-sampled frame image data, then the global detection model is used to detect the target object in the non-sampled frame image, and the area occupied by the target object is determined as the fourth recognition area; the fourth recognition area is the sensitive area of ​​the non-sampled frame image data.

9. The method as described in claim 8, characterized in that, The step of determining whether a new target object appears in the first sampled frame image data after the non-sampled frame image data specifically includes: The first identification region is matched with the first prediction region to determine whether there is a first identification region that does not match the first prediction region.

10. The method according to any one of claims 1-5, characterized in that, The training method for the global detection model specifically includes: The first detection model is trained using a multi-class labeled dataset to obtain a trained backbone network; the first detection model consists of a backbone network and a third detection head, wherein the third detection head is a multi-class detection head; The parameters of the trained backbone network are fixed, and the second detection model is trained using the dataset labeled with the first object to obtain the trained first detection head; the second detection model consists of the trained backbone network and the first detection head. The parameters of the trained backbone network are fixed, and the third detection model is trained using the dataset labeled with the second object to obtain the trained second detection head; the third detection model consists of the trained backbone network and the second detection head.

11. A computer device, characterized in that, A global detection model is used to detect target objects, including a first object and a second object. The global detection model includes a backbone network, a first detection head, and a second detection head. The backbone network is used for feature extraction from image data. The first detection head is a binary classification detection head used to detect the first object, and the second detection head is a binary classification detection head used to detect the second object. The model includes: The feature extraction module is used to extract features from the acquired image data using the backbone network to obtain image features; The first detection module is used to detect the first object and the second object based on the image features using the first detection head and the second detection head respectively, and to determine the area occupied by the first object and the second object as the first recognition area; the first recognition area is the sensitive area of ​​the image data; wherein, the first object is a face; and the second object is a license plate; The second determining module is used to: determine a preliminary missed region in the global detection model based on motion estimation of target objects detected in video frame image data preceding the video frame image data; and use a local detection model to verify whether a target object exists in the preliminary missed region to determine a second identification region. The preliminary missed region represents the area occupied by the undetected target object. The second identification region is a sensitive region of the video frame image data.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 10.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 10.

Images