Target detection method, vehicle-mounted device and storage medium

Abstract

An embodiment of the present application provides a target detection method, a vehicle-mounted device and a storage medium, the method comprising: acquiring a visible light image and an infrared image of a target scene, aligning the visible light image and the infrared image to obtain a visible light registered image and an infrared registered image; forming data points according to the target object sizes contained in the visible light registered image and the infrared registered image to obtain an anchor frame data set; clustering based on the anchor frame data set to determine a cluster center, and determining a target anchor frame according to the data points corresponding to the cluster center; acquiring an image to be identified of a scene to be identified, slidingly traversing the image to be identified based on the target anchor frame to extract image features from image areas corresponding to the target anchor frame respectively, and obtaining a target detection result of whether the image to be identified contains a target object according to the image features.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a target detection method and device, a vehicle-mounted device, and a computer-readable storage medium. Background Art

[0002] In recent years, assisted driving with the goal of reducing accidents has gradually become a social need. Visible light-based target detection technology has been widely used in automobile assisted driving and has achieved certain development. However, due to the extremely complex actual road conditions, such as at night, haze, insufficient light and other conditions where visibility is poor and the field of vision is limited, the effect of visible light will be greatly affected, the driver's visual distance will be significantly shortened, and driving safety will be greatly affected and threatened. Infrared imaging technology can convert the infrared radiation emitted by the scene itself into a visible thermal image. Since different parts of an object have different radiation characteristics, the infrared imaging system can distinguish the various parts of the scene and then convert them into visible grayscale images. The acquisition of infrared images does not depend on external light, has strong environmental applicability and strong anti-interference ability. Therefore, the vehicle-mounted system that combines visible light images and infrared images for target detection can make up for the shortcomings of visible light in driving scenes, help drivers clearly observe surrounding vehicles and pedestrians, discover safety warning information in advance, and improve driving safety.

[0003] However, known target detection algorithms are usually only applicable to processing a single visible light image or infrared image. When they are directly applied to processing visible light images and infrared images at the same time, the use of sliding windows for target detection will lead to window redundancy and high time complexity, resulting in low recall and accuracy of target detection. Summary of the invention

[0004] In order to solve the existing technical problems, the present application provides a target detection method, a vehicle-mounted device and a computer-readable storage medium that can combine visible light images and infrared images for target detection and effectively improve the recall rate and accuracy of target detection.

[0005] To achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a target detection method, comprising:

[0007] Acquire a visible light image and an infrared image of a target scene, and register the visible light image and the infrared image to obtain a visible light registered image and an infrared registered image;

[0008] Forming data points according to the target object sizes contained in the visible light registration image and the infrared registration image to obtain an anchor frame data set;

[0009] Performing clustering based on the anchor frame data set to determine a cluster center, and determining a target anchor frame according to the data points corresponding to the cluster center;

[0010] An image to be identified of a scene to be identified is obtained, and the image to be identified is traversed by sliding based on the target anchor frame to extract image features from image areas corresponding to the target anchor frame respectively, and a target detection result is obtained according to the image features to determine whether the image to be identified contains a target object.

[0011] In a second aspect, an embodiment of the present application provides a vehicle-mounted device, comprising a processor, an image acquisition device connected to the processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the image acquisition device is used to synchronously acquire visible light images and infrared images corresponding to each moment of the driving scene; when the computer program is executed by the processor, the target detection method as described in any embodiment of the present application is implemented.

[0012] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, the target detection method as described in any embodiment of the present application is implemented.

[0013] In the above embodiment, the visible light image and the infrared image of the target scene are registered to obtain the visible light registered image and the infrared registered image. Based on the size of the target object contained in the visible light registered image and the infrared registered image, an anchor frame data set is formed to determine the target anchor frame. The image to be identified of the scene to be identified is traversed by sliding through the target anchor frame to respectively extract the image features of the image area corresponding to the target anchor frame to achieve target detection. In this way, the anchor frame size is optimized and the target detection is performed based on the target object contained in the visible light registered image and the infrared registered image. On the one hand, the same target detection algorithm can be simultaneously applied to target detection combining visible light images and infrared images, and the target detection advantage of visible light images is still maintained in scenes with sufficient visible light. In addition, the target detection of infrared images can be used to make up for the detection requirements in driving scenes with insufficient visible light, helping the driver to clearly observe the surrounding vehicles and pedestrians, discover safety warning information in advance, and improve driving safety. On the other hand, the optimized anchor frame size is used for sliding traversal to identify whether the image to be identified contains the target object, which can reduce window redundancy and improve recognition efficiency, improve the target detection capability of the scene, and effectively increase the recall rate and accuracy of target detection.

[0014] In the above embodiments, the vehicle-mounted device, the computer-readable storage medium and the corresponding target detection method embodiments belong to the same concept, and thus have the same technical effects as the corresponding target detection method embodiments, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is an architecture diagram of an optional application scenario of a target detection method in an embodiment;

[0016] Figure 2 is a flow chart of a target detection method in one embodiment;

[0017] Figure 3 is a flow chart of a target detection method in another embodiment;

[0018] Figure 4 This is a schematic diagram of distance calculation in one embodiment;

[0019] Figure 5 A schematic diagram of the architecture of an initial neural network model in one embodiment;

[0020] Figure 6 is a flowchart of a target detection method in an optional specific example;

[0021] Figure 7 A comparison diagram of target detection effects in an embodiment;

[0022] Figure 8 is a schematic diagram of the structure of a target detection device in an embodiment;

[0023] Fig. 9 Schematic diagram of the structure of the vehicle-mounted equipment in one embodiment. DETAILED DESCRIPTION

[0024] The technical solution of the present application is further elaborated in detail below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0025] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0026] In the following description, the expression "some embodiments" is involved, which describes a subset of all possible embodiments. It should be noted that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0027] In the following description, the terms "first, second, third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second, third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] See also Figure 1 , which is a schematic diagram of an optional application scenario of the target detection method provided in an embodiment of the present application, wherein the vehicle-mounted device includes a main control system 13, and an image acquisition device 11 that is communicatively connected to the main control system 13, and the main control system 13 mainly includes a memory, a processor, a display that is communicatively connected to the processor, and an input device, etc. The image acquisition device 11 is used to acquire infrared images and visible light images of the same target scene at the same time, for example, the image acquisition device 11 may refer to a dual-light camera that simultaneously acquires visible light images and infrared images, or an infrared shooting module installed at a specific position on the vehicle that can synchronously and at the same viewing angle acquire infrared images of the same target scene and a visible light shooting module that acquires visible light images. The vehicle-mounted equipment is installed on the vehicle. When the vehicle is driving on the lane, the image acquisition equipment collects infrared images around the vehicle in real time to form an infrared image of the driving scene, and visible light images around the vehicle in real time to form a visible light image of the driving scene, and sends them to the main control system 13. The main control system 13 aligns the infrared image and the visible light image of the same target scene at the same time to obtain corresponding visible light registered images and infrared registered images, and uses the visible light registered images and infrared registered images as images to be recognized. The target anchor frame is used to slide and traverse the image to be recognized to extract image features, and the extracted image features are used to determine whether the image to be recognized contains a target object.

[0029] See also Figure 2 , which is a target detection method provided by an embodiment of the present application, can be applied to Figure 1 The vehicle-mounted device shown, the target detection method comprises the following steps:

[0030] S101, acquiring a visible light image and an infrared image of a target scene, and registering the visible light image and the infrared image to obtain a registered visible light image and a registered infrared image.

[0031] The target scene refers to the scene that contains the target object to be identified. For target detection in a vehicle driving scene, the target scene refers to the scene corresponding to the environment in which the vehicle is driving. For example, when a vehicle is driving on a lane, the target scene refers to the scene formed by a certain range of areas in front, behind, left and right of the vehicle's position during the driving process on the lane. Visible light images refer to pictures or video data obtained by imaging based on the light bands of the electromagnetic spectrum that can be perceived by the human eye. Infrared images refer to pictures or video data obtained by imaging based on infrared radiation.

[0032] Registering the visible light image and the infrared image may refer to registering the visible light image and the infrared image collected at the same time and the same target scene using an edge image matching algorithm so that the common target in the visible light image and the infrared image can be consistent in spatial orientation. In some optional embodiments, registering a visible light image and an infrared image may include the following steps: synchronously collecting a visible light image and an infrared image of a target scene through a visible light and infrared dual-light camera; graying the visible light image and the infrared image to obtain a visible light grayscale image and an infrared grayscale image; respectively extracting edge information of the visible light grayscale image and the infrared grayscale image to obtain edge images of the visible light grayscale image and the infrared grayscale image; using the edge image of the infrared grayscale image to perform a horizontal and vertical translation traversal of the visible light grayscale image, and after each translation, respectively calculating the degree of fit between the edge image of the infrared grayscale image in the current translation state and the edge image of the visible light grayscale image, and after the traversal is completed, statistically obtaining the horizontal and vertical translation values ​​of the infrared grayscale image when the degree of fit is maximum; translating the infrared image based on the translation value, and registering the translated infrared image with the visible light image to obtain a registered visible light registration image and an infrared registration image.

[0033] Taking the target scene as a vehicle driving scene as an example, the visible light image and infrared image of the target scene refer to the visible light image and infrared image taken for the environment in which the vehicle is currently driving, such as the visible light image and infrared image taken for a certain range of areas in front, behind, left and right of the vehicle on the lane when the vehicle is driving on the lane. By obtaining the visible light image and infrared image of the scene in which the vehicle is located during the vehicle driving process, other vehicles, people or objects that are close to the vehicle around the vehicle can be captured in the visible light image and infrared image of the corresponding driving scene, so that the visible light image and infrared image of the corresponding driving scene can be detected to identify whether the corresponding driving scene contains the target object. According to the recognition result of the target object, the risk of collision that may occur during vehicle driving can be predicted and an early warning can be issued.

[0034] Target detection refers to detecting whether the visible light image and infrared image of the target scene contain targets such as people, vehicles or other objects. Optionally, the target detection result may include the target object category and its position. The target object category refers to the category of different targets contained in the visible light image and infrared image of the target scene, wherein the number of target object categories, category names, etc. may be preset, such as the target object category includes people and vehicles; or the target object category is divided into pedestrians, cyclists, and electric bike riders for people, and is divided into cars, buses, trucks and other categories of vehicles for vehicles. The target object position refers to the position of the identified target object in the corresponding image.

[0035] S103, forming data points according to the target object sizes contained in the visible light registration image and the infrared registration image to obtain an anchor frame data set.

[0036] The target object size can be obtained by annotating the target objects contained in the visible light registration image and the infrared registration image. In an optional specific example, the target object size includes the width and height of the target object, and the width and height of each target object are used as a data point to form an anchor frame data set. By acquiring visible light images and infrared images of the target scene to form a training sample set, the visible light images and infrared images collected at the same time in the training sample set are registered, and the target objects contained in the registered visible light registration images and infrared registration images are annotated to obtain the target object size to form an anchor frame data set. By increasing the number of visible light registration images and infrared registration images used as training samples, the target object categories included can be made more comprehensive and accurate, so that the anchor frame size determined based on the target object size contained in the visible light registration images and infrared registration images can accurately and quickly perform sliding detection on the visible light image and the infrared image at the same time.

[0037] S105 , clustering is performed based on the anchor frame data set to determine a cluster center, and a target anchor frame is determined according to the data points corresponding to the cluster center.

[0038] Clustering refers to the process of dividing a collection of physical or abstract objects into multiple classes consisting of similar objects. The clusters generated by clustering are a collection of data objects that are similar to objects in the same cluster and different from objects in other clusters. Clustering is performed based on the anchor frame data set to determine the cluster center, and the target anchor frame is determined according to the data points corresponding to the cluster center. In this way, by dividing the similar data points in the anchor frame data set into the same cluster and determining the cluster center corresponding to each cluster, each cluster center can be used as the representative data point of its cluster, and other data points in the same cluster have similarities with the corresponding cluster center. The target anchor frame is determined according to the width and height of the target object represented by the data point corresponding to the cluster center, so that the size of the target anchor frame can be used to more accurately and quickly detect all target objects with similar sizes.

[0039] S107, obtaining an image to be identified of a scene to be identified, slidingly traversing the image to be identified based on the target anchor frame to extract image features from image areas corresponding to the target anchor frame respectively, and obtaining a target detection result of whether the image to be identified contains a target object based on the image features.

[0040] The scene to be identified refers to a scene to which a target detection method is applied to collect its corresponding image for detection to identify whether it contains a target object. Generally, the scene to be identified refers to a scene with an environment similar to that of the target scene. Taking the application of the target detection method to a vehicle-mounted device as an example, the scene to be identified and the target scene both refer to a vehicle driving scene. The image to be identified refers to a visible light image and an infrared image obtained by collecting the scene to be identified for detecting whether it contains a target object. In some optional embodiments, the image to be identified refers to a visible light registration image and an infrared registration image obtained by registering the visible light image and the infrared image collected based on the scene to be identified for detecting whether it contains a target object.

[0041] Taking the scene to be identified as a driving scene as an example, the image to be identified is a driving scene image, specifically including a visible light image and an infrared image corresponding to the driving scene. The traversing the image to be identified through the target anchor frame to extract image features, and obtaining a target detection result of whether the image to be identified contains a target object according to the image features, may refer to the vehicle-mounted device acquiring a driving scene image, sliding the driving scene image in sequence through the target anchor frame to perform target detection to determine whether the driving scene image contains target objects such as people and vehicles, and when it is determined that the driving scene image contains target objects such as people and vehicles, the category of the target object and the position of the target object in the driving scene image are determined, and the target detection result is formed and output according to the category and position of the target object contained in the driving scene image. Optionally, the target detection result also includes the size of the target object, and the vehicle-mounted device performs target detection on the driving scene image, and when it is determined that the driving scene image contains target objects such as people and vehicles, the category of the target object, the position of the target object in the driving scene image, and the size of the target object in the driving scene image are determined, and the target detection result is formed and output according to the category, position, and size of the target object contained in the driving scene image.

[0042] Among them, the target anchor frame is determined after optimization through clustering based on the size of the target object contained in the visible light image and the infrared image of the target scene. In the process of determining the target anchor frame, the initial anchor frame is determined based on the target object size in the visible light registered image and the infrared registered image obtained after registration based on the visible light image and the infrared image collected by the target scene. The common target in the visible light image and the infrared image can be consistent in spatial orientation through registration, and then the size of the optimized target anchor frame is determined according to the target object size in the visible light registered image and the infrared registered image, so that the optimized target anchor frame can adapt to the detection accuracy and detection speed of the visible light image and the infrared image collected in the same scene.

[0043] In the above embodiment, the visible light image and the infrared image of the target scene are registered to obtain the visible light registered image and the infrared registered image. Based on the size of the target object contained in the visible light registered image and the infrared registered image, an anchor frame data set is formed to determine the target anchor frame. The image to be identified of the scene to be identified is traversed by sliding through the target anchor frame to respectively extract the image features of the image area corresponding to the target anchor frame to achieve target detection. In this way, the anchor frame size is optimized and the target detection is performed based on the target object contained in the visible light registered image and the infrared registered image. On the one hand, the same target detection algorithm can be simultaneously applied to target detection combining visible light images and infrared images, and the target detection advantage of visible light images is still maintained in scenes with sufficient visible light. In addition, the target detection of infrared images can be used to make up for the detection requirements in driving scenes with insufficient visible light, helping the driver to clearly observe the surrounding vehicles and pedestrians, discover safety warning information in advance, and improve driving safety. On the other hand, the optimized anchor frame size is used for sliding traversal to identify whether the image to be identified contains the target object, which can reduce window redundancy and improve recognition efficiency, improve the target detection capability of the scene, and effectively increase the recall rate and accuracy of target detection.

[0044] See also Figure 3 The step S105, performing clustering based on the anchor frame data set to determine a cluster center, and determining a target anchor frame according to the data point corresponding to the cluster center, includes:

[0045] S1051, determining an initial cluster center according to the anchor frame data set;

[0046] S1052, calculating a first distance between each data point in the anchor frame data set and the initial cluster center, and updating the initial cluster center according to the first distance to obtain an updated cluster center;

[0047] S1057, determining a target anchor frame according to the data points corresponding to the finally updated cluster center.

[0048] Optionally, random sampling can be used to select any data point from the anchor frame data set as the initial cluster center, and the first distance between each data point in the anchor frame data set and the initial cluster center is calculated. Based on the calculation result of the first distance, a data point with a relatively large first distance can be selected as a new cluster center, and the first distance between each data point in the anchor frame data set and the new cluster center is calculated. The cluster center is again updated based on the calculation result of the first distance, and the cycle is repeated until multiple cluster centers are selected as the final updated cluster centers. For the sake of ease of description, the repeated cycle until the final updated cluster center is selected for the anchor frame data set is regarded as one clustering iteration.

[0049] The calculation of the first distance between two data points can be determined according to the ratio of the intersection and union of the anchor boxes corresponding to the two data points, where Xi represents any data point in the anchor box data set, and D(Xi) represents the distance between the data point and the cluster center, where D(Xi) = 1-IOU(box, centroid), such as Figure 4 As shown, it is a schematic diagram of the ratio of the intersection and union of the anchor boxes represented by two data points respectively.

[0050] In the above embodiment, the cluster center of the anchor frame data set is determined by clustering, and the target anchor frame is determined by the data points corresponding to the cluster center. Clustering can divide the data points with similarity into the same cluster, so that the target anchor frame determined based on the cluster center can adapt to the detection of the target objects corresponding to the data points in the same cluster.

[0051] Optionally, after calculating the first distance between each data point in the anchor frame data set and the initial cluster center, and updating the initial cluster center according to the first distance to obtain the updated cluster center, the step S1052 further includes:

[0052] S1053, taking the current cluster center as the initialization centroid, performing clustering division according to the distance between each data point in the anchor frame data set and the initialization centroid, and determining a cluster set;

[0053] The current cluster center may refer to the final updated cluster center obtained after a clustering iteration based on the initial anchor frame data set. The current cluster center is used as the initialization centroid, and clustering is performed according to the distance between each data point in the anchor frame data set and the initialization centroid, and each data point is respectively assigned to the set to which the initialization centroid belongs relatively closest to it, and the cluster set is determined. In other optional embodiments, the current cluster center may also refer to the updated cluster center obtained by updating the previously determined cluster center each time during a clustering iteration of the anchor frame data set, and this is used as the initialization centroid for judgment, which can be used as a condition for determining the termination of a clustering iteration.

[0054] S1054, obtaining an updated anchor frame data set according to the newly added data points formed by the target object size;

[0055] When the data points included in the anchor frame dataset change, such as when new data points are added to the anchor frame dataset, an updated anchor frame dataset is obtained according to the newly added data points formed by the size of the target object.

[0056] S1055, calculating a second distance between each data point in the updated anchor frame data set and the current cluster center, updating the current cluster center according to the second distance, and determining an updated centroid according to the current updated cluster center;

[0057] Based on the second distance between each data point in the updated anchor frame data set and the current cluster center determined according to the previous anchor frame data set, according to the calculation result of the second distance, select the data point with a relatively large second distance as the new cluster center, and classify each data point into the set to which the cluster center relatively closest to it belongs, and these cluster centers can be used as the updated centroid determined based on the updated anchor frame data set. Among them, the calculation of the second distance between two data points can also be determined according to the ratio IOU of the intersection and union of the anchor frames respectively represented by the two data points. Xi represents any data point in the anchor frame data set, D(Xi) represents the distance between the data point and the cluster center, and D(Xi) = 1-IOU(box, centroid).

[0058] S1056: If the distance between the initialization centroid and the update centroid meets a set condition, the cluster center corresponding to the initialization centroid is used as the final cluster center.

[0059] Calculate the distance between the initialization centroid and the update centroid. If the distance between the initialization centroid and the update centroid meets the set conditions, such as being less than the preset distance value, the cluster center corresponding to the initialization centroid can be used as the final cluster center. That is, the cluster center determined based on the previous anchor frame data set can be used as the final cluster center.

[0060] Optionally, the target detection method further includes:

[0061] If the distance between the initialization centroid and the updated centroid does not satisfy the set condition, the step of determining the initial cluster center according to the anchor frame data set is returned to re-cluster the updated anchor frame data set to re-determine the target anchor frame.

[0062] If the distance between the initial centroid and the updated centroid does not meet the set conditions, such as being greater than or equal to the preset distance value, then the process returns to the re-clustering step based on the updated anchor frame data set, re-clustering each data point in the updated anchor frame data set, and re-classifying each data point into the set to which the cluster center is relatively closest, and re-determining the target anchor frame. For the updated anchor frame data set, clustering based on the initial centroid to determine whether the target anchor frame needs to be recalculated is beneficial to reducing the amount of calculation while keeping the target anchor frame optimal relative to the current target detection solution.

[0063] In the above embodiment, the cluster center of the anchor frame data set is determined by clustering, the target anchor frame is determined by the data point corresponding to the cluster center, and the anchor frame data set can be updated and upgraded by adding new data points, and the cluster set is re-determined for the updated anchor frame data frame to ensure that the optimal cluster center value can always be obtained according to the update of the anchor frame data set, so that the target anchor frame determined based on the cluster center can adapt to the detection of target objects corresponding to each data point in the same cluster, so that the optimized target anchor frame can improve the detection accuracy and efficiency.

[0064] In some embodiments, the step S1052 of calculating a first distance between each data point in the anchor frame data set and the initial cluster center, and updating the initial cluster center according to the first distance to obtain an updated cluster center includes:

[0065] Taking each data point in the anchor frame data set as a target data point, determining a first distance between the target data point and the initial cluster center according to an intersection-over-union ratio of the target data point and the data point corresponding to the initial cluster center;

[0066] The target data point whose first distance meets a preset condition is selected as the updated cluster center.

[0067] Among them, the intersection-over-union ratio of the target data point and the data point corresponding to the initial cluster center refers to the ratio of the intersection and union (IOU) of the two anchor boxes represented by the target data point and the data point corresponding to the initial cluster center. The preset condition may refer to various pre-set rules. Selecting the target data point whose first distance meets the preset condition as the updated cluster center may refer to determining the data point that meets the corresponding rule in the first distance between each calculated data point and the corresponding cluster center according to the preset rule as the target data point, and using the target data point as the updated cluster center. Optionally, the first distance meeting the preset condition may refer to one of the following: the first distance is greater than a preset threshold, the calculation result of the first distance is ranked in the front preset position, and the first distance is greater than the preset threshold and is ranked in the front preset position in the current first distance calculation result.

[0068] In the above embodiment, the distance calculation formula between the two data points is determined by calculating the intersection and union ratio of the anchor boxes corresponding to the two data points. In this way, when calculating the distance between the anchor boxes corresponding to the two data points in the anchor box data set, the overlap, distance and scale between the two anchor boxes are taken into consideration, so that the distance calculation between the anchor boxes corresponding to the two target objects with overlap can be realized, and the comprehensive measurement of the overlap ratio, graphic distance and shape similarity (aspect ratio) between the two anchor boxes is realized.

[0069] In some embodiments, the step S107, obtaining an image to be identified of a scene to be identified, slidingly traversing the image to be identified based on the target anchor frame to extract image features from image regions corresponding to the target anchor frame, and obtaining a target detection result of whether the image to be identified contains a target object according to the image features, includes:

[0070] Acquire an image to be recognized of a scene to be recognized; wherein the image to be recognized includes a visible light registered image and an infrared registered image obtained by registering the collected visible light image and infrared image of the scene to be recognized;

[0071] Input the image to be identified into the target detection model obtained by training the initial neural network model;

[0072] The target detection model uses the target anchor frame to traverse the image to be identified to extract image features, and obtains a target detection result of whether the image to be identified contains a target object based on the image features.

[0073] The initial neural network model can adopt an image recognition model of a known target detection algorithm. Known target detection algorithms may include traditional target detection algorithms and detection algorithms based on deep learning. Traditional target detection algorithms can use artificially designed feature extractors, such as HOG (Histogram of Oriented Gradient, HOG) and SIFT (Scale-invariant feature transform, SIFT) features commonly used in pedestrian detection and general target detection, to extract target features in the image, and then extract features of the candidate area of ​​the image by sliding the detection window in sequence, and finally use commonly used support vector machines (SVM), Adaboost and other classifiers to determine whether there is a target in the candidate area. Detection algorithms based on deep learning can autonomously learn and improve based on deep convolutional neural networks and target data sets, such as R-CNN (Region-Convolutional Neural Networks), fast R-CNN, faster R-CNN, etc. These methods first use the regional candidate network to extract candidate target information, and then use the detection network to predict and identify the location and category of the candidate target; such as SSD (single shot Multibox detector), YOLO, etc. These methods can directly obtain the category and location information of the target through the network to form an end-to-end target detection algorithm.

[0074] The inventors of the present application have found in their research that the image recognition model of the known target detection algorithm is only suitable for performing target detection on a single visible light image or a single infrared image, however, a single visible light image cannot meet the target detection requirements in scenes with insufficient lighting, and a single infrared image is obviously inferior to a visible light image when used for target detection in scenes with sufficient lighting conditions. How to make the image recognition model of the known target detection algorithm maintain good performance when performing target detection on both visible light images and infrared images is the technical pain point and difficulty that the inventors of the present application hope to solve this time. In an embodiment of the present application, a visible light image and an infrared image of the target scene are obtained for registration, an anchor frame data set is formed according to the target object size contained in the visible light registration image and the infrared registration image obtained after the registration, and a size-optimized target anchor frame is determined according to the anchor frame data set. In actual application, the vehicle-mounted equipment collects visible light images and infrared images of the scene to be identified, and aligns the collected visible light images and infrared images to obtain visible light registered images and infrared registered images as the images to be identified of the scene to be identified. The target detection model after training the initial neural network model uses a target anchor frame to slide over the image to be identified, and image features are extracted from the image area corresponding to the target anchor frame to identify whether the image to be identified contains the target object based on the image features. In this way, after the image recognition model of the known target detection algorithm is trained, it can maintain good performance when combining visible light images and infrared images for target detection.

[0075] In an optional specific example, the initial neural network model uses the infiray_yolov5s convolutional neural network model. Figure 5 , the network structure of yolov5s consists of four parts, namely the input layer Input, the backbone network Backbone, the neck layer Neck, and the output layer Output. The input layer Input is mainly used to perform image enhancement preprocessing on the image to be identified, including but not limited to image flipping, adaptive scaling, mosaic data enhancement, and random brightness changes; the backbone network Backbone is used to extract the feature information of the target and input the feature map into the neck structure Neck; the neck structure Neck fuses the feature maps of different backbone layers, fuses the shallow network with the deep network, and enhances the fusion ability of network features; the output layer Output is used to generate bounding boxes and predict the category of the target. The visible light registration image and the infrared registration image obtained by registering the visible light image and the infrared image of the target scene form training samples. After training the initial neural network model, the target detection model is obtained.

[0076] In the above embodiment, by simultaneously collecting visible light images and infrared images of the scene to be identified and aligning them as the images to be identified, and by optimizing the size of the anchor frame, the image recognition model of the known target detection algorithm can be trained to maintain good performance when performing target detection on both visible light images and infrared images at the same time, thereby effectively compatibly using the advantages of using visible light images and infrared images for target detection in scenes with sufficient lighting and scenes with insufficient lighting, thereby improving the accuracy and efficiency of the target detection results.

[0077] Optionally, the step S103, forming data points according to the target object size contained in the visible light registration image and the infrared registration image to obtain an anchor frame data set, includes:

[0078] Obtaining an image set according to the visible light registration image and the infrared registration image;

[0079] Using random sampling to divide the image set into a training data set and a test data set according to a set ratio;

[0080] Forming data points according to the target object sizes contained in the visible light registration image and the infrared registration image in the training data set, obtaining an anchor frame data set;

[0081] Before inputting the image to be identified into the target detection model obtained by training the initial neural network model, the method further includes:

[0082] Constructing an initial neural network model, inputting the training data set into the initial neural network model for iterative training, and obtaining an initial target detection model;

[0083] Inputting the test data set into the initial target detection model for validation;

[0084] If the validation fails, the training data set is updated, and the process returns to the step of inputting the training data set into the initial neural network model for iterative training;

[0085] If the verification is successful, the currently obtained initial target detection model is used as the target detection model obtained after training.

[0086] The target scene refers to various similar scenes including the target object to be identified in the scene to be identified. For example, if the scene to be identified is a driving scene, the visible light image and infrared image of the target scene are the visible light image and infrared image collected by the vehicle under different driving environments. An image set is obtained by registering the visible light image and infrared image collected at the same time and in the same target scene, and the image set is divided into a training data set and a test data set. The anchor frame size is optimized based on the visible light registration image and infrared registration image included in the training data set, and the initial neural network model is trained based on the training data set until the training reaches the corresponding termination condition to obtain an initial target detection model, and then the initial target detection model is validated using the visible light registration image and infrared registration image included in the test data set. The target detection performance of the trained target detection model can be further ensured through validation.

[0087] In an optional specific example, mean Average Precision (MAP) can be used as an evaluation indicator to evaluate the detection performance of the target detection model for combining visible light images and infrared images for target detection. MAP is the average of the mean precisions of all categories in the data set, and AP is the mean precision of a category. For the i-th category, different IOU thresholds are selected, and the calculation method of the mean precision is shown in the following formula 1:

[0088]

[0089] The geometric meaning of Formula 1 is the area enclosed by the curve formed by precision and recall and the horizontal axis.

[0090] The calculation method of average precision MAP is shown in the following formula 2:

[0091]

[0092] The MAP comparison between target detection based on single visible light image and target detection combining visible light image and infrared image is shown in the following table:

[0093] Model MAP Single visible light 87.69％ Visible light image + infrared image 90.13％

[0094] In the above embodiment, the initial neural network model uses the Yolov5s network as an example. On the vehicle-mounted data set, the MAP of the target detection model obtained by using both visible light image and infrared image data is greatly improved compared to using only a single visible light data set.

[0095] In order to have a more comprehensive understanding of the target detection method provided in the embodiment of the present application, please refer to Figure 6, the target detection method is applied to a vehicle-mounted device, and the target scene and the scene to be identified are both vehicle driving scenes, to illustrate the target detection method. The vehicle-mounted device includes a visible light and infrared dual-light camera and a main control system, wherein the main control system includes a memory and a processor.

[0096] S11, collecting visible light images and infrared images of the driving scene; wherein the vehicle-mounted device can collect the visible light images and infrared images of the vehicle driving scene through the visible light and infrared dual-light cameras mounted on the vehicle, and can use the target detection and annotation tools to annotate them, thereby obtaining the visible light images and infrared images at the same time;

[0097] S12, registering the visible light image and the infrared image collected at the same time to obtain a registered visible light registered image and an infrared registered image; wherein a matching algorithm based on an edge image may be used to realize data registration of the visible light image and the infrared image, so that the common target in the visible light image and the infrared image can be consistent in spatial orientation;

[0098] S13, using a random sampling method to divide the collected driving scene image data set into independent training sets and test sets according to a certain ratio;

[0099] S14, determining the target anchor box by a clustering algorithm; wherein, a set of initial candidate boxes (Anchorboxes) with fixed width and height can be determined to form an anchor box data set according to the target object sizes contained in the visible light registration image and the infrared registration image contained in the training set; clustering the anchor box data set to determine the optimal target anchor box size, mainly including the following steps:

[0100] (a) Calculate the width and height of all target objects contained in the visible light registration images and infrared registration images in the training set and use them as a data set.

[0101] (b) Randomly select a point from the data set as the first cluster center C1.

[0102] (c) Calculate the distance D(xi) between each point xi in the data set and the nearest cluster center among the selected cluster centers. D(xi) = 1-IOU(box, centroid), where IOU is the intersection over union, that is, the ratio of the intersection and union of two anchor boxes.

[0103] (d) Select a new data point as the new cluster center. The selection principle is: the point with larger D(x) has a greater probability of being selected as the cluster center.

[0104] (e) Repeat c and d until k cluster centers are selected.

[0105] (f) Using these k cluster centers as the initial centroids, for each point in the data set, calculate its distance to each initial centroid (the distance calculation method is the same as the above IOU calculation). The point is classified into the set to which the initial centroid it is closest to belongs.

[0106] (g) Divide all data points into corresponding clusters, there are k clusters in total, and recalculate the centroid of each cluster.

[0107] (h) If the distance between the new centroid and the original centroid changes greatly, steps f and g need to be iterated. If the distance between the newly calculated centroid and the original centroid is less than a certain set threshold (indicating that the position of the recalculated centroid does not change much and tends to be stable), the final cluster center value is obtained, that is, the value of the cluster target anchor box.

[0108] Among them, when the data points included in the data set change, if a new data point is added to obtain an updated data set, the new data point can be processed by executing steps (f) to (h) to preliminarily determine whether the value of the current target anchor box is still applicable. If not, the target anchor box is re-determined by executing steps (a) to (h). Optionally, when the data points included in the data set change, if a new data point is added to obtain an updated data set, the target anchor box can also be directly re-determined by executing steps (a) to (h).

[0109] S15, using the training set to train the initial neural network model, during the training process, the neural network model uses the size-optimized target anchor frame to detect the training sample image until a trained target detection model is obtained, and using the test set to verify the detection ability of the target detection model until a final target detection model is obtained;

[0110] S16, real-time collection of visible light images and infrared images during vehicle driving, and the corresponding visible light registration images and infrared registration images are formed by registration as the images to be identified and input into the target detection model, and target detection is performed through the target detection model to output the target object type, position and size target detection results.

[0111] In the above embodiments, target detection is achieved by combining visible light images and infrared images in vehicle driving scenes, which effectively improves the target detection accuracy and recall rate, increases the target recognition rate of vehicle-mounted equipment, and improves driving safety. Figure 7 As shown in FIG. 7-1, a comparison diagram of target detection effect based on a single visible light image and a target detection effect based on a combination of a visible light image and an infrared image is shown in FIG. Figure 7It can be seen that the target detection method provided in the embodiment of the present application, combined with visible light images and infrared images, can effectively improve the target detection effect.

[0112] See also Figure 8 On the other hand, an embodiment of the present application provides a target detection device, including: an acquisition module 211, used to acquire a visible light image and an infrared image of a target scene, align the visible light image and the infrared image, and obtain a visible light registered image and an infrared registered image; a labeling module 212, used to form data points according to the target object sizes contained in the visible light registered image and the infrared registered image, and obtain an anchor frame data set; a clustering module 213, used to perform clustering based on the anchor frame data set to determine a cluster center, and determine a target anchor frame according to the data points corresponding to the cluster center; a detection module 214, used to acquire an image to be identified of a scene to be identified, slide traverse the image to be identified based on the target anchor frame, so as to extract image features from the image area corresponding to the target anchor frame respectively, and obtain a target detection result of whether the image to be identified contains a target object according to the image features.

[0113] Optionally, the clustering module 213 is specifically used to determine an initial cluster center based on the anchor frame data set; calculate a first distance between each data point in the anchor frame data set and the initial cluster center, update the initial cluster center according to the first distance to obtain an updated cluster center; and determine a target anchor frame based on the data points corresponding to the finally updated cluster center.

[0114] Optionally, the clustering module 213 is further used to use the current cluster center as the initialization centroid, perform clustering according to the distance between each data point in the anchor frame data set and the initialization centroid, and determine a cluster set; obtain an updated anchor frame data set according to the data points formed by the newly added target object size; calculate the second distance between each data point in the updated anchor frame data set and the current cluster center, update the current cluster center according to the second distance, and determine an updated centroid according to the current updated cluster center; if the distance between the initialization centroid and the update centroid meets the set conditions, use the cluster center corresponding to the initialization centroid as the final cluster center.

[0115] Optionally, the clustering module 213 is further configured to return to the step of determining an initial cluster center according to the anchor frame data set if the distance between the initialization centroid and the update centroid does not satisfy the set condition.

[0116] Optionally, the clustering module 213 is also used to take each data point in the anchor frame data set as a target data point, and determine a first distance between the target data point and the initial cluster center according to an intersection-and-union ratio of the target data point and the data point corresponding to the initial cluster center; and select the target data point whose first distance meets a preset condition as the updated cluster center.

[0117] Optionally, the detection module 214 is specifically used to obtain an image to be identified of a scene to be identified; wherein the image to be identified includes a visible light registered image and an infrared registered image obtained by registering the collected visible light image and infrared image of the scene to be identified; the image to be identified is input into a target detection model obtained after training an initial neural network model; the target detection model is used to traverse the image to be identified using the target anchor frame to extract image features, and a target detection result of whether the image to be identified contains a target object is obtained according to the image features.

[0118] Optionally, the detection module 214 is further used to obtain an image set based on the visible light registration image and the infrared registration image; to divide the image set into a training data set and a test data set according to a set ratio by random sampling; to form data points according to the target object sizes contained in the visible light registration image and the infrared registration image in the training data set to obtain an anchor frame data set; the target detection device also includes a training module, which is used to construct an initial neural network model, input the training data set into the initial neural network model for iterative training, and obtain an initial target detection model; input the test data set into the initial target detection model for verification; if the verification fails, update the training data set, and return to the step of inputting the training data set into the initial neural network model for iterative training; if the verification passes, use the currently obtained initial target detection model as the target detection model obtained after training.

[0119] It should be noted that: the target detection device provided in the above embodiment only uses the division of the above program modules as an example to illustrate the process of real-time detection and identification of target objects that may have a collision risk with the vehicle during vehicle driving. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the method steps described above. In addition, the target detection device provided in the above embodiment belongs to the same concept as the target detection method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0120] See also Fig. 9On the other hand, an embodiment of the present application further provides a vehicle-mounted device, including a processor 21, an image acquisition device 11 connected to the processor 21, a memory 22, and a computer program stored in the memory 22 and executable by the processor 21, wherein the image acquisition device 11 is used to synchronously acquire visible light images and infrared images corresponding to each moment of the driving scene; when the computer program is executed by the processor 21, the target detection method described in any embodiment of the present application is implemented. Among them, the processor 21 includes an ECU processing unit; the image acquisition device 11 can be a vehicle-mounted visible light and infrared dual-light camera. The processor 21 is installed in the vehicle cab, receives the visible light image and infrared image captured by the visible light and infrared dual-light camera, and implements the target detection method described in the embodiment of the present application.

[0121] Optionally, the vehicle-mounted device further comprises a display warning module connected to the processor 21. The display warning device can present a simulated video with a final result to the customer, and provide sound, light and / or picture warnings for possible collision risks.

[0122] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned target detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is such as a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0123] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling an intelligent terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0125] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A target detection method, characterized in that: include: Acquire a visible light image and an infrared image of a target scene, and register the visible light image and the infrared image to obtain a visible light registered image and an infrared registered image; Forming data points according to the target object sizes contained in the visible light registration image and the infrared registration image to obtain an anchor frame data set; Performing clustering based on the anchor frame data set to determine a cluster center, and determining a target anchor frame according to the data points corresponding to the cluster center; Acquire an image to be recognized of a scene to be recognized; wherein the image to be recognized includes a visible light registered image and an infrared registered image obtained by registering the collected visible light image and infrared image of the scene to be recognized; Input the image to be identified into the target detection model obtained by training the initial neural network model; The target detection model uses the target anchor frame to traverse the image to be identified to extract image features, and obtains a target detection result of whether the image to be identified contains a target object based on the image features.

2. The target detection method according to claim 1, characterized in that: The step of clustering the anchor frame data set to determine a cluster center, and determining a target anchor frame according to the data point corresponding to the cluster center, includes: Determining an initial clustering center according to the anchor frame data set; Calculating a first distance between each data point in the anchor frame data set and the initial cluster center, and updating the initial cluster center according to the first distance to obtain an updated cluster center; The target anchor frame is determined according to the data points corresponding to the finally updated cluster center.

3. The target detection method according to claim 2, characterized in that: The step of calculating a first distance between each data point in the anchor frame data set and the initial cluster center, and updating the initial cluster center according to the first distance to obtain an updated cluster center, further comprises: Taking the current cluster center as the initialization centroid, performing clustering division according to the distance between each data point in the anchor frame data set and the initialization centroid, and determining a cluster set; Obtaining an updated anchor frame dataset according to the newly added data points formed by the target object size; Calculating a second distance between each data point in the updated anchor frame data set and the current cluster center, updating the current cluster center according to the second distance, and determining an updated centroid according to the current updated cluster center; If the distance between the initialization centroid and the update centroid meets the set condition, the cluster center corresponding to the initialization centroid is used as the final cluster center.

4. The target detection method according to claim 3, characterized in that: Also includes: If the distance between the initialization centroid and the update centroid does not satisfy the set condition, return to the step of determining the initial cluster center according to the anchor frame data set.

5. The target detection method according to claim 2, characterized in that: The calculating a first distance between each data point in the anchor frame data set and the initial cluster center, and updating the initial cluster center according to the first distance to obtain an updated cluster center, comprises: Taking each data point in the anchor frame data set as a target data point, determining a first distance between the target data point and the initial cluster center according to an intersection-over-union ratio of the target data point and the data point corresponding to the initial cluster center; The target data point whose first distance meets a preset condition is selected as the updated cluster center.

6. The target detection method according to claim 1, characterized in that: The forming of data points according to the target object sizes contained in the visible light registration image and the infrared registration image to obtain an anchor frame data set includes: Obtaining an image set according to the visible light registration image and the infrared registration image; Using random sampling to divide the image set into a training data set and a test data set according to a set ratio; Forming data points according to the target object sizes contained in the visible light registration image and the infrared registration image in the training data set, obtaining an anchor frame data set; Before inputting the image to be identified into the target detection model obtained by training the initial neural network model, the method further includes: Constructing an initial neural network model, inputting the training data set into the initial neural network model for iterative training, and obtaining an initial target detection model; Inputting the test data set into the initial target detection model for validation; If the validation fails, the training data set is updated, and the process returns to the step of inputting the training data set into the initial neural network model for iterative training; If the verification is successful, the currently obtained initial target detection model is used as the target detection model obtained after training.

7. A vehicle-mounted device, comprising a processor, an image acquisition device connected to the processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the image acquisition device is used to synchronously acquire visible light images and infrared images corresponding to each moment of a driving scene; When the computer program is executed by the processor, the target detection method according to any one of claims 1 to 6 is implemented.

8. The vehicle-mounted device according to claim 7, characterized in that: The processor includes an ECU processing unit; the image acquisition device is a vehicle-mounted visible light and infrared dual-light camera.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the target detection method according to any one of claims 1 to 6 is implemented.

Images