A method, device and storage medium for processing obstructed objects

Abstract

The present invention discloses a method, device and storage medium for processing an obstructed target. The method comprises collecting current image data of a road; processing the current image data based on a pre-constructed target processing model to obtain the category and position frame of the target; wherein the construction of the target processing model comprises: obtaining historical image data of the road; manually marking the category, position frame and occlusion status of the target in the historical image data according to preset marking rules; performing augmentation processing and data encoding on the marked historical image data to generate a historical image data set; training a YOLOv5 neural network through the historical image data set to construct a target processing model; the occlusion status comprises no occlusion, slight occlusion and severe occlusion; the present invention can solve the technical problem of accurate identification of the category and position frame of the target, and facilitate more accurate decision-making for subsequent assisted driving.

Description

Technical Field

[0001] The present invention relates to a method, a device and a storage medium for processing an obstructed target object, and belongs to the technical field of driving assistance. Background Art

[0002] As the process of electric and intelligent vehicles continues to deepen, people's expectations are rising, software has changed the attributes of cars, and the continuous evolution of autonomous driving has gradually liberated the attention of drivers, until it is completely free of human intervention. Vehicles have also changed from simple means of transportation to mobile private spaces. The realization of autonomous driving mainly relies on various sensors to perceive the environment around the vehicle, and to assess the danger through processing technologies such as target detection, recognition and tracking. Among them, target detection technology is the basis of autonomous driving technology. Through the camera, the image data of the road ahead is obtained in real time, and the category and location of various targets in the image are preliminarily determined, providing information for more accurate recognition and control of subsequent vehicles.

[0003] In the task of detecting road targets, there are various occlusion scenarios. In summary, there are two types of occlusion: (1) mutual occlusion between detection targets; (2) occlusion of detection targets by interfering objects. In common detection tasks, there are also two ways to deal with severely occluded targets: (1) in general scenarios, there is no need to pay attention to severely occluded targets, so samples of severely occluded targets are directly discarded and not learned; (2) in some more special scenarios, such as dense crowd detection and livestock count, severely occluded targets must be planned into the samples, and the usual practice is to improve performance by increasing the number of training samples of occlusion categories. Due to the special nature of ADAS scenarios, better use of the already scarce computing resources allows the system to more quickly respond to targets with collision risks, and occluded targets on the road are usually not risk targets for direct collision with the vehicle, so it is necessary to appropriately adjust the learning tendency of the network. For targets with more severe occlusion, the requirements are lowered, and the target category and rough position can be determined without missed detection. For targets with no occlusion or slight occlusion, it is necessary to accurately determine the target category and its precise position to facilitate more accurate decision-making later. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method, device and storage medium for processing objects with occlusion, so as to solve the technical problem of accurately identifying the category and position frame of the object.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a method for processing an obstructed object, comprising:

[0007] Collect current image data of the road;

[0008] Process the current image data based on the pre-built object processing model to obtain the category and position frame of the object;

[0009] Wherein, the construction of the target object processing model includes:

[0010] Obtain historical image data of roads;

[0011] Manually annotate the category, location frame and occlusion status of the target objects in the historical image data according to the preset annotation rules;

[0012] Perform augmentation processing and data encoding on the annotated historical image data to generate a historical image dataset;

[0013] The YOLOv5 neural network is trained using historical image datasets to build a target object processing model.

[0014] The occlusion status includes no occlusion, slight occlusion and severe occlusion.

[0015] Optionally, the collecting of the current image data of the road includes collecting video data of different lighting conditions, different scenes, and different viewing angles in real time, and extracting frame segments containing different targets from the video data as the collecting of the current image data of the road.

[0016] Optionally, the augmentation processing includes image cropping, image flipping, Mixup data enhancement and Mosaic data enhancement.

[0017] Optionally, the data encoding includes normal encoding of historical image data with no occlusion or slight occlusion, and normal encoding of historical image data with severe occlusion and adding a severe occlusion flag.

[0018] Optionally, the training of the YOLOv5 neural network through the historical image dataset to construct the target object processing model includes:

[0019] Divide the historical image dataset into training set, test set and validation set;

[0020] Input the training samples in the training set into the YOLOv5 neural network to obtain the predicted output;

[0021] Calculate the total loss L based on the predicted output of the YOLOv5 neural network and the validation set Det :

[0022]

[0023] Among them, L obj is the target confidence loss, Lclass is the target category loss, L box is the target position box loss, is the target category loss containing occlusion state, The loss of the position frame of the target object in the occluded state. are weight parameters respectively, y is the heavy occlusion flag, if the occluded target is heavily occluded, y = 1, otherwise y = 0;

[0024] Using the total loss L Det Perform back-propagation iteration on the network parameters of the YOLOv5 neural network;

[0025] The YOLOv5 neural network is updated according to the iterative network parameters, and the updated YOLOv5 neural network is tested according to the test set. If the preset requirements are met, the updated YOLOv5 neural network is used as the target object processing model.

[0026] Optionally, the target category loss and the target category loss with occlusion state are calculated and obtained by a cross-entropy loss function; the cross-entropy loss function is:

[0027]

[0028] Among them, C is the target category loss, x is the test sample, n is the number of test samples in the test set, y is the annotation information of the target category, and a is the prediction information of the target category output by the YOLOv5 neural network.

[0029] Optionally, the target position box loss and the target position box loss with occlusion state are calculated and obtained by using a CIoU Loss loss function; the CIoU Loss loss function is:

[0030]

[0031]

[0032]

[0033]

[0034] Among them, L CIoU is the target location box loss, A is the labeled target location box, and B is the target location box predicted by the YOLOv5 neural network output; ρ 2 (b,b gt ) is the center point b of the marked target position frame A and the center point b of the predicted target position frame B gtThe Euclidean distance between the marked target position box and the predicted target position box B is c, w and h are the width and height of the marked target position box, respectively. gt and h gt are the width and height of the predicted target location box respectively;

[0035] The target confidence loss is represented by the IoU value.

[0036] In a second aspect, the present invention provides a method and device for processing an obstructed object, the device comprising:

[0037] A data acquisition module, used to collect current image data of the road;

[0038] A data processing module, used to process the current image data based on a pre-built target object processing model to obtain the category and position frame of the target object;

[0039] Wherein, the construction of the target object processing model includes:

[0040] Obtain historical image data of roads;

[0041] Manually annotate the category, location frame and occlusion status of the target objects in the historical image data according to the preset annotation rules;

[0042] Perform augmentation processing and data encoding on the annotated historical image data to generate a historical image dataset;

[0043] The YOLOv5 neural network is trained using historical image datasets to build a target object processing model.

[0044] The occlusion status includes no occlusion, slight occlusion and severe occlusion.

[0045] In a third aspect, the present invention provides a method and device for processing an obstructed object, including a processor and a storage medium;

[0046] The storage medium is used to store instructions;

[0047] The processor is used to operate according to the instructions to execute the steps of any of the methods described above.

[0048] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of any of the above-mentioned methods.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The present invention provides a method for processing occluded targets. Through unbalanced learning, for severely occluded targets, the loss of their categories is calculated normally, the loss weight of their categories is reduced, and the back propagation of the position frame loss is limited. This reduces the problem of large differences and high learning difficulty caused by treating severe occlusion and no occlusion or mild occlusion as the same category, and ultimately improves the category and position frame information of no occlusion or mild occlusion targets, while also preventing the omission of severely occluded targets, facilitating more accurate decision-making for subsequent assisted driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of a method for processing an obstructed target provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0053] Embodiment 1:

[0054] like Figure 1 As shown, an embodiment of the present invention provides a method for processing an obstructed target, comprising the following steps:

[0055] (1) Collect current image data of the road;

[0056] Collecting the current image data of the road includes collecting video data of different lighting conditions, different scenes, and different viewing angles in real time, and extracting frame segments containing different targets from the video data as the current image data of the road.

[0057] (2) Processing the current image data based on the pre-built object processing model to obtain the category and location box of the object;

[0058] The construction of the target object processing model includes:

[0059] Obtain historical image data of roads;

[0060] Manually annotate the category, location frame and occlusion status of the target objects in the historical image data according to the preset annotation rules;

[0061] Perform augmentation processing and data encoding on the annotated historical image data to generate a historical image dataset;

[0062] The YOLOv5 neural network is trained using historical image datasets to build a target object processing model.

[0063] The occlusion states include no occlusion, slight occlusion, and severe occlusion.

[0064] Specific:

[0065] 1. Augmentation processing includes image cropping, image flipping, Mixup data enhancement and Mosaic data enhancement.

[0066] 2. Data encoding includes normal encoding of historical image data with no occlusion or slight occlusion, normal encoding of historical image data with severe occlusion, and adding a severe occlusion flag.

[0067] 3. Train the YOLOv5 neural network through historical image datasets to build a target object processing model including:

[0068] 3.1. Divide the historical image dataset into training set, test set and validation set;

[0069] 3.2. Input the training samples in the training set into the YOLOv5 neural network to obtain the predicted output;

[0070] 3.3. Calculate the total loss L based on the predicted output of the YOLOv5 neural network and the validation set Det :

[0071]

[0072] Among them, L obj is the target confidence loss, L class is the target category loss, L box is the target position box loss, is the target category loss containing occlusion state, The loss of the position frame of the target object in the occluded state. are weight parameters respectively, y is the heavy occlusion flag, if the occluded target is heavily occluded, y = 1, otherwise y = 0;

[0073] 3.4. Using total loss L Det Perform back-propagation iteration on the network parameters of the YOLOv5 neural network;

[0074] 3.5. Update the YOLOv5 neural network according to the iterated network parameters, and test the updated YOLOv5 neural network according to the test set. If it meets the preset requirements, the updated YOLOv5 neural network will be used as the target object processing model.

[0075] 3.3.1. The target category loss and the target category loss with occlusion state are calculated by the cross-entropy loss function; the cross-entropy loss function is:

[0076]

[0077] Among them, C is the target category loss, x is the test sample, n is the number of test samples in the test set, y is the annotation information of the target category, and a is the prediction information of the target category output by the YOLOv5 neural network.

[0078] 3.3.2. The target position box loss and the target position box loss with occlusion state are calculated by CIoU Loss loss function; CIoU Loss loss function is:

[0079]

[0080]

[0081]

[0082]

[0083] Among them, L CIoU is the target location box loss, A is the labeled target location box, and B is the target location box predicted by the YOLOv5 neural network output; ρ 2 (b,b gt ) is the center point b of the marked target position frame A and the center point b of the predicted target position frame B gt The Euclidean distance between the marked target position box and the predicted target position box B is c, w and h are the width and height of the marked target position box, respectively. gt and h gt are the width and height of the predicted target location box respectively;

[0084] 3.3.3. The target confidence loss is expressed as IoU value.

[0085] Embodiment 2:

[0086] An embodiment of the present invention provides a method and device for processing an obstructed target, the device comprising:

[0087] A data acquisition module, used to collect current image data of the road;

[0088] A data processing module, used to process the current image data based on a pre-built target object processing model to obtain the category and position frame of the target object;

[0089] The construction of the target object processing model includes:

[0090] Obtain historical image data of roads;

[0091] Manually annotate the category, location frame and occlusion status of the target objects in the historical image data according to the preset annotation rules;

[0092] Perform augmentation processing and data encoding on the annotated historical image data to generate a historical image dataset;

[0093] The YOLOv5 neural network is trained using historical image datasets to build a target object processing model.

[0094] The occlusion states include no occlusion, slight occlusion, and severe occlusion.

[0095] Embodiment three:

[0096] Based on the first embodiment, the embodiment of the present invention further provides a method and device for processing an obstructed target, including a processor and a storage medium;

[0097] The storage medium is used to store instructions;

[0098] The processor is configured to operate according to the instructions to execute the steps according to any one of the above methods.

[0099] Embodiment 4:

[0100] Based on the first embodiment, the embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of any of the above methods are implemented.

[0101] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for processing an obstructed object, characterized in that: include: Collect current image data of the road; Process the current image data based on the pre-built object processing model to obtain the category and position frame of the object; Wherein, the construction of the target object processing model includes: Obtain historical image data of roads; Manually annotate the category, location frame and occlusion status of the target objects in the historical image data according to the preset annotation rules; Perform augmentation processing and data encoding on the annotated historical image data to generate a historical image dataset; The YOLOv5 neural network is trained using historical image datasets to build a target object processing model. The occlusion state includes no occlusion, slight occlusion and severe occlusion; The method of training the YOLOv5 neural network through the historical image data set to construct the target object processing model includes: Divide the historical image dataset into training set, test set and validation set; Input the training samples in the training set into the YOLOv5 neural network to obtain the predicted output; Calculate the total loss L based on the predicted output of the YOLOv5 neural network and the validation set Det : Among them, L obj is the target confidence loss, L class is the target category loss, L box is the target position box loss, is the target category loss containing occlusion state, The loss of the position frame of the target object in the occluded state. are weight parameters respectively, y is the heavy occlusion flag, if the occluded target is heavily occluded, y = 1, otherwise y = 0; Using the total loss L Det Perform back-propagation iteration on the network parameters of the YOLOv5 neural network; Update the YOLOv5 neural network according to the iterated network parameters, and test the updated YOLOv5 neural network according to the test set. If the preset requirements are met, the updated YOLOv5 neural network is used as the target object processing model; The target category loss and the target category loss with occlusion state are calculated by the cross-entropy loss function; the cross-entropy loss function is: Among them, C is the target category loss, x is the test sample, n is the number of test samples in the test set, y is the annotation information of the target category, and a is the prediction information of the target category output by the YOLOv5 neural network; The target position box loss and the target position box loss with occlusion state are calculated by CIoU Loss function; the CIoU Loss function is: Among them, L CIoU is the target location box loss, A is the labeled target location box, and B is the target location box predicted by the YOLOv5 neural network output; ρ 2 (b,b gt ) is the center point b of the marked target position frame A and the center point b of the predicted target position frame B gt The Euclidean distance between the marked target position box and the predicted target position box B is c, w and h are the width and height of the marked target position box, respectively. gt and h gt are the width and height of the predicted target location box respectively; The target confidence loss is represented by the IoU value.

2. The method for processing an obstructed object according to claim 1, characterized in that: The collecting of the current image data of the road includes collecting video data of different lighting conditions, different scenes, and different viewing angles in real time, and extracting frame segments containing different targets from the video data as the collected current image data of the road.

3. The method for processing an obstructed object according to claim 1, characterized in that: The augmentation processing includes image cropping, image flipping, Mixup data enhancement and Mosaic data enhancement.

4. The method for processing an obstructed object according to claim 1, characterized in that: The data encoding includes normal encoding of historical image data with no occlusion or slight occlusion, and normal encoding of historical image data with severe occlusion and adding a severe occlusion flag.

5. A method and device for processing an obstructed object, characterized in that: The device adopts the steps of the method according to any one of claims 1 to 4, and the device comprises: A data acquisition module, used to collect current image data of the road; A data processing module, used to process the current image data based on a pre-built target object processing model to obtain the category and position frame of the target object; Wherein, the construction of the target object processing model includes: Obtain historical image data of roads; Manually annotate the category, location frame and occlusion status of the target objects in the historical image data according to the preset annotation rules; Perform augmentation processing and data encoding on the annotated historical image data to generate a historical image dataset; The YOLOv5 neural network is trained using historical image datasets to build a target object processing model. The occlusion status includes no occlusion, slight occlusion and severe occlusion.

6. A method and device for processing an obstructed object, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images