Target detection method, device, storage medium and equipment

Abstract

The present application provides a target detection method, device, storage medium and equipment. In this method, a target image sequence is used as a detection cycle. The first frame image in the detection cycle is preliminarily detected by a first target detection model to identify the regional positions of all targets on the image. After that, the distant target area is divided into multiple categories, and these multiple categories are assigned to the remaining images in the sequence, so as to cut out a distant target area map, and then perform target detection on the distant target area map to obtain distant target information. At the same time, a frame is selected from the remaining images according to the timeline of the target image sequence, and a near target area map is cropped from it, and then target detection is performed on the near target area map to obtain near target information. In this way, while achieving stable detection of distant and near targets, the computing consumption is reduced and the computing efficiency is improved, thereby meeting real-time requirements and reducing the demand for chip computing power.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target detection method, device, storage medium and equipment. Background Art

[0002] At present, the technology of on-board road condition perception based on camera 2D images is widely used in urban NOA (Navigate on Autopilot, navigation-assisted autonomous driving), and the accurate and effective perception of distant and nearby dynamic and static targets has become one of the main indicators for improving intelligent driving performance. The existing target detection scheme based on monocular vision mainly performs preliminary detection on the input image through a neural network model with a large parameter amount, and then performs regional cutouts on the image based on the preliminary detection results and then sends it to another neural network model for refined detection to obtain distant target information and nearby target information. However, when there are more targets in the preliminary detection results, the time taken for cutouts and subsequent detection will become longer, which often cannot meet the real-time requirements, and the actual vehicle-mounted chip computing power requirements are high. Summary of the invention

[0003] The purpose of this application is to provide a target detection method, device, storage medium and equipment, aiming to solve the problem that the target detection method based on monocular vision in the related technology is difficult to meet the real-time requirements and has high requirements on chip computing power.

[0004] In a first aspect, the present application provides a target detection method, comprising: processing a first frame image in a target image sequence collected by a vehicle-mounted camera through a first target detection model to obtain preliminary detection information; the preliminary detection information includes regional position information of all targets in the first frame image; obtaining a distant target area according to the preliminary detection information, dividing the distant target area into multiple categories, and assigning the multiple categories to the remaining images in the target image sequence, cropping a distant target area map corresponding to the assigned category from the remaining images, and selecting a frame image from the remaining images according to the timeline of the target image sequence, and cropping a near target area map from the selected image; performing target detection on the distant target area map and the near target area map to obtain distant target information and near target information; the distant target information includes the posture information of the distant targets in the target image sequence; the near target information includes the posture information of the near targets in the target image sequence.

[0005] In the above implementation process, the target image sequence is taken as a detection cycle, and the first frame image in the target image sequence is preliminarily detected by the first target detection model to identify the regional positions of all targets on the image. After that, the distant target area is divided into multiple categories, and these multiple categories are assigned to the remaining images of the target image sequence, so as to cut out the distant target area map, and then perform target detection on the distant target area map to obtain the distant target information. At the same time, according to the timeline of the target image sequence, a frame is selected from the remaining images, and a near target area map is cropped out from it, and then target detection is performed on the near target area map to obtain the near target information. In this way, while achieving stable detection of distant and near targets, the computing consumption is reduced and the computing efficiency is improved, thereby meeting the real-time requirements and reducing the demand for chip computing power.

[0006] Furthermore, in some examples, the number of image frames of the target image sequence is set according to a perception frame rate of the vehicle-mounted camera.

[0007] In the above implementation process, the number of image frames within the detection cycle is dynamically determined according to the actual perception frame rate of the vehicle-mounted camera, which effectively avoids missing targets in certain frames, thereby improving the accuracy and reliability of target detection.

[0008] Further, in some examples, the method of obtaining a distant target area based on the preliminary detection information and dividing the distant target area into multiple categories includes: intercepting the distant target area in the first frame image based on the regional position information of all distant targets in the first frame image; and clustering the intercepted distant target area using a clustering algorithm to obtain multiple categories.

[0009] In the above implementation process, the distant target area is intercepted based on the regional position information of the distant target output by the first target detection model. After all the distant target areas are intercepted, the clustering algorithm is used to divide these distant targets to be detected into multiple categories. In this way, the detection tasks of the distant target area are reasonably allocated, thereby effectively improving the detection efficiency and reducing the computing power required for target detection.

[0010] Furthermore, in some examples, the clustering algorithm is a K-means clustering algorithm; the K value of the K-means clustering algorithm is a value obtained by subtracting one from the number of image frames of the target image sequence.

[0011] In the above implementation process, the K-means clustering algorithm is used to cluster and merge distant target areas. The number of categories is consistent with the number of remaining image frames in the target image sequence except the first frame, so that when the next image performs inference, each frame has a distant target detection task, thereby effectively improving the rationality and reliability of the allocation of distant target detection tasks.

[0012] Furthermore, in some examples, the preliminary detection information also includes the confidence of all targets in the first frame image; the assigning of the multiple categories to the remaining images in the target image sequence includes: calculating the cumulative confidence of the category based on the confidence of the distant targets contained in each category; arranging the multiple categories in ascending order according to the cumulative confidence; and assigning the multiple categories to the remaining images in the target image sequence in sequence according to the arrangement order of the multiple categories, so that each frame of the remaining images is assigned to a category.

[0013] In the above implementation process, a specific method for allocating distant target detection tasks is provided, and distant targets with low confidence are detected first, thereby ensuring the long-term stability and accuracy of the detection results over the entire time series.

[0014] Further, in some examples, selecting a frame of image from the remaining images according to the timeline of the target image sequence includes: determining at least one frame of image corresponding to an intermediate moment among the remaining images according to the timeline of the target image sequence; and selecting a frame of image from the at least one frame of image corresponding to the intermediate moment.

[0015] In the above implementation process, a near target area map is captured from at least one frame of image corresponding to the intermediate moment to perform near target detection, thereby effectively reducing computing consumption and improving detection efficiency while ensuring stable detection of near targets.

[0016] Further, in some examples, the target detection is performed on the distant target area map and the near target area map to obtain distant target information and near target information, including: performing target detection on the distant target area map through a second target detection model to obtain distant target information, and performing target detection on the near target area map through a third target detection model to obtain near target information; wherein, the second target detection model is trained based on a first training data set; the first training data set includes a plurality of distant target sample images and a label corresponding to each distant target sample image; the label corresponding to the distant target sample image includes the true value of a specified attribute of the distant target in the distant target sample image; the third target detection model is trained based on the second training data set; the second training data set includes a plurality of near target sample images and a label corresponding to each near target sample image; the label corresponding to the near target sample image includes the true value of a specified attribute of the near target in the near target sample image.

[0017] In the above implementation process, two neural network models are trained, one dedicated to detecting distant targets and the other for detecting nearby targets, thereby effectively improving the accuracy of perception.

[0018] In a second aspect, the present application provides a target detection device, including: a preliminary detection module, used to process the first frame image in the target image sequence collected by the vehicle-mounted camera through a first target detection model to obtain preliminary detection information; the preliminary detection information includes the regional position information of all targets in the first frame image; a region cropping module, used to obtain a distant target region according to the preliminary detection information, divide the distant target region into multiple categories, and assign the multiple categories to the remaining images in the target image sequence, crop out a distant target region map corresponding to the assigned category from the remaining images, and select a frame of image from the remaining images according to the timeline of the target image sequence, and crop out a near target region map from the selected image; a target detection module, used to perform target detection on the distant target region map and the near target region map to obtain distant target information and near target information; the distant target information includes the posture information of the distant target in the target image sequence; the near target information includes the posture information of the near target in the target image sequence.

[0019] In a third aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.

[0021] In a fifth aspect, the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method as described in any one of the first aspects.

[0022] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flow chart of a target detection method provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the overall process of a near and far target detection solution based on time sequence allocation provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a training data source for a neural network model specifically used for detecting distant targets provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a training data source for a neural network model specifically used for detecting nearby targets provided in an embodiment of the present application;

[0029] Figure 5 A block diagram of a target detection device provided in an embodiment of the present application;

[0030] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] As described in the background technology, the target detection method based on monocular vision in the related art has the problem of being difficult to meet the real-time requirements and requiring high chip computing power. Based on this, the embodiment of the present application provides a new target detection solution to solve the above problems.

[0034] Next, the embodiments of the present application are introduced:

[0035] like Figure 1 As shown, Figure 11 is a flow chart of a target detection method provided in an embodiment of the present application, wherein the method is applied to an automated driving control unit (ADCU) of a vehicle, hereinafter referred to as a controller.

[0036] The method comprises:

[0037] Step 101: Processing the first frame of a target image sequence collected by a vehicle-mounted camera through a first target detection model to obtain preliminary detection information; the preliminary detection information includes the regional position information of all targets in the first frame of the image;

[0038] The vehicle-mounted camera mentioned in this step can be a monocular camera or other types of cameras. During the driving process of the vehicle, the surrounding environment can be perceived in real time based on the vehicle-mounted camera. The first target detection model mentioned in this step can be a neural network model for predicting all targets of the input image frame, which can be a model with a large parameter amount. In the present embodiment, a target image sequence is composed of N frames of images continuously collected by the vehicle-mounted camera, and the first frame of the target image sequence is input into the first target detection model. The first target detection model identifies the regional position of the distant target and the near target on the image. With the help of the preliminary detection information output by the first target detection model, the controller can have a preliminary understanding of the target distribution in the current field of view. In addition, generally speaking, the first target detection model can also identify the approximate categories of distant targets and near targets.

[0039] In some embodiments, the number of image frames of the target image sequence mentioned in this step is set according to the perception frame rate of the vehicle-mounted camera. In other words, the number of image frames of the target image sequence is dynamically determined according to the perception frame rate of the actual vehicle-mounted camera, which can effectively avoid the situation where the target is missed in some frames, thereby improving the accuracy and reliability of target detection.

[0040] Step 102: acquiring a distant target region according to the preliminary detection information, dividing the distant target region into a plurality of categories, assigning the plurality of categories to the remaining images in the target image sequence, cropping a distant target region map corresponding to the assigned categories from the remaining images, selecting a frame of image from the remaining images according to the timeline of the target image sequence, and cropping a near target region map from the selected image;

[0041] In this embodiment, based on the phenomenon that the time interval between consecutive frames is short and the image content will not change suddenly, a task allocation scheme with the number of image frames of the target image sequence as a whole detection cycle is proposed. First, the first frame image is processed by the first target detection model to obtain the regional position information of all targets. Then, since the distribution of distant targets in the camera imaging field of view is random and their regional ranges may be more divergent, the distant target areas are divided into multiple categories and assigned to the next N-1 frames of images for prediction. Compared with the traditional scheme of processing all targets in each frame, this embodiment can effectively reduce the computing consumption, and since the time interval between consecutive frames is short, the accuracy and reliability of the detection results will not be reduced.

[0042] In some embodiments, the step of obtaining the distant target area according to the preliminary detection information and dividing the distant target area into multiple categories may include: intercepting the distant target area in the first frame image according to the regional position information of all distant targets in the first frame image; clustering the intercepted distant target area using a clustering algorithm to obtain multiple categories. In other words, the controller may intercept the distant target area based on the regional position information of the distant target output by the first target detection model, and after intercepting all the distant target areas, use a clustering algorithm to divide the distant targets to be detected into multiple categories. In this way, a reasonable allocation of detection tasks for distant target areas is achieved, thereby effectively improving detection efficiency and reducing the computing power required for target detection.

[0043] Optionally, the clustering algorithm mentioned above can be a K-means clustering algorithm; the K value of the K-means clustering algorithm is the value obtained by subtracting one from the number of image frames of the target image sequence. The K-means clustering algorithm is a widely used clustering analysis algorithm, the core idea of ​​which is to divide the data set into K clusters so that the data points in the same cluster have high similarity, while the data points between different clusters have low similarity. In implementation, the controller sets the K value to N-1 according to the number of image frames N of the target image sequence, and clusters all the intercepted distant target areas with this K value, thereby dividing N-1 categories. In this way, the number of categories is consistent with the number of remaining image frames of the target image sequence except the first frame, and the multiple categories divided can be evenly distributed to the remaining images, that is, each image corresponds to one category, thereby effectively improving the rationality and reliability of the allocation of distant target detection tasks.

[0044] Of course, in other embodiments, other methods may be used to divide the distant target area into multiple categories according to the needs of specific scenarios. For example, in some scenarios, the distant targets to be detected may be divided into designated K categories according to the principle that the area tends to be consistent.

[0045] Further, in some embodiments, the aforementioned preliminary detection information may also include the confidence of all targets in the first frame image; then the assignment of the multiple categories to the remaining images in the target image sequence mentioned in this step may include: calculating the cumulative confidence of the category according to the confidence of the distant targets contained in each category; arranging the multiple categories in ascending order according to the cumulative confidence; and assigning the multiple categories to the remaining images in the target image sequence in sequence according to the arrangement order of the multiple categories, so that each frame of the remaining images is assigned to a category. That is to say, the result output by the first target detection model also includes the confidence of all targets, which is an important indicator to measure the degree of confidence of the model in its prediction results. After dividing the distant target area into multiple categories, the controller may average the confidence of the distant targets contained in each category to obtain the cumulative confidence of the corresponding category, and then perform target detection in the next K frame time in the order from small to large according to the value of the cumulative confidence of each category to obtain the perception output for the distant target, that is, the distant target information. In this way, multiple categories are sorted in order from small to large, so that the controller can give priority to detecting distant targets with low confidence, thereby ensuring the long-term stability and accuracy of the detection results over the entire time series.

[0046] Since the imaging area of ​​the near target in the camera imaging field of view is relatively fixed, there is no need to cluster the near target area. In this embodiment, according to the timeline of the target image sequence, a frame is selected from the next K frames of images, and a near target area map is cropped from it according to the area position information of the near target indicated in the preliminary detection information, and then target detection is performed on the near target area map to obtain the perception output for the near target, that is, the near target information.

[0047] In some embodiments, the step of selecting a frame of image from the remaining images according to the timeline of the target image sequence mentioned in this step may include: determining at least one frame of image corresponding to the intermediate moment in the remaining images according to the timeline of the target image sequence; and selecting a frame of image from the at least one frame of image corresponding to the intermediate moment. That is, in the next K frames of images, the image corresponding to the intermediate moment can be determined. If the number of images corresponding to the intermediate moment is one frame, the near target area of ​​the image corresponding to the intermediate moment is intercepted for near target detection. If the number of images corresponding to the intermediate moment is two frames, one of the two frames is selected as the target image, and the near target area of ​​the target image is intercepted for near target detection. Since the image corresponding to the intermediate moment can better represent the characteristics of the entire target image sequence, the result obtained by performing target detection on the near target area map can reflect the near target detection result of the entire target image sequence to a certain extent. In this way, compared with the method of processing all targets in each frame in the traditional scheme, the calculation consumption can be effectively reduced and the detection efficiency can be improved.

[0048] Step 103, perform target detection on the distant target area map and the near target area map to obtain distant target information and near target information; the distant target information includes the position and posture information of the distant targets in the target image sequence; the near target information includes the position and posture information of the near targets in the target image sequence.

[0049] This step means: according to the preliminary detection results output by the first target detection model, all distant target areas are divided into multiple categories and assigned to the next image to be detected in the current detection cycle, so as to crop out a distant target area map, and at the same time select a frame from these images to be detected, from which a near target area map is cropped, and then the distant target area map and the near target area map are respectively refined to obtain distant target information and near target information. The distant target information here includes the posture information of the distant target, such as the position and orientation of the distant target in the 3D world, and correspondingly, the near target information includes the posture information of the near target, such as the position and orientation of the near target in the 3D world, that is, the target detection performed on the distant target area map and the near target area map can be 3D target detection. The preliminary detection information output by the first target detection model includes the regional position information of distant and near targets, as well as preliminary classification information of distant and near targets, such as general categories such as vehicles, people, and obstacles. The distant target information and the near target information can respectively include details of the categories of distant and near targets, such as engineering vehicles, tricycles, trucks, the elderly, children, road posts, stones, etc., and can also include details of the status of distant targets, such as the distance from the vehicle, the length, width, and height in the 3D world, etc.

[0050] In some embodiments, this step may include: performing target detection on the distant target area map through a second target detection model to obtain distant target information, and performing target detection on the near target area map through a third target detection model to obtain near target information; wherein, the second target detection model is trained based on the first training data set; the first training data set includes multiple distant target sample images and a label corresponding to each distant target sample image; the label corresponding to the distant target sample image includes the true value of the specified attribute of the distant target in the distant target sample image; the third target detection model is trained based on the second training data set; the second training data set includes multiple near target sample images and a label corresponding to each near target sample image; the label corresponding to the near target sample image includes the true value of the specified attribute of the near target in the near target sample image.

[0051] That is to say, a neural network model specifically used for detecting distant targets, i.e., the second target detection model, is pre-trained to perform target detection on the distant target area map, thereby ensuring stable and accurate detection of distant targets. The training data of the second target detection model can be collected by the vehicle camera, and the label is the specified attribute of the distant target in the picture, such as the target's subdivision category, the 2D frame position in the picture, the length, width, height, direction, and distance from the vehicle in the 3D real world. These values ​​can be obtained by manual measurement through instruments and annotated on the picture. In this way, the second target detection model can learn to extract features related to the specified attributes of the distant target from the input image during the training process, and perform target classification, regression, and distance estimation based on the extracted features, thereby obtaining distant target information, including the category, position, posture, etc. of the distant target. Similarly, a neural network model specifically used for detecting near targets, i.e., the third target detection model, is pre-trained to perform target detection on the near target area map, thereby ensuring stable and accurate detection of near targets. The training data of the second target detection model can also be collected by the vehicle camera, and the label is the true value of the specified attribute of the near target annotated manually. In this way, different detection tasks are dynamically performed through the three detection models of the first target detection model, the second target detection model and the third target detection model, thereby effectively improving the accuracy of the target detection results. In addition, the algorithms used by the second target detection model and the third target detection model can be any 3D target detection algorithm, such as Mono3D, Deep3DBox, etc. The specific training process and working principle can be referred to the introduction of the corresponding algorithm in the relevant technology, and this application will not be repeated here.

[0052] In the embodiment of the present application, a target image sequence collected by a vehicle-mounted camera is used as a detection cycle. The first frame image in the target image sequence is preliminarily detected by a first target detection model to identify the regional positions of all targets on the image. Based on this preliminary detection result, the distant target area is divided into multiple categories, and these multiple categories are assigned to the remaining images of the target image sequence, thereby intercepting a distant target area map, and then performing target detection on the distant target area map to obtain distant target information. At the same time, a frame is selected from the remaining images according to the timeline of the target image sequence, and a near target area map is cropped from it, and then target detection is performed on the near target area map to obtain near target information. In this way, by assigning detection tasks to the intercepted areas of distant targets and near targets, while ensuring stable detection of distant and near targets, the computing efficiency is improved, thereby meeting real-time requirements and reducing the demand for chip computing power.

[0053] In order to explain the solution of this application in more detail, a specific embodiment is introduced below:

[0054] This embodiment provides a near and far target detection solution based on time sequence allocation. This solution is applied to the vehicle's intelligent driving system to perceive the surrounding environment and output vehicle control decisions based on the perception results to assist the driver in driving safely. The overall process of this embodiment is as follows Figure 2 As shown, this embodiment scheme uses three neural network models, among which the first detection network 21 is a neural network model with a large number of parameters, which is used to detect all targets in the input image frame; according to the output of the first detection network 21, a preliminary understanding of the target distribution in the current field of view can be obtained; the second detection network 22 is a neural network model specially used to detect distant targets, and its training data source is as follows Figure 3 As shown, Figure 3 In the figure, region 31 is one of the multiple distant target regions in the image, which represents the region in the image of some pedestrians and some vehicles that are far away from the vehicle. In implementation, the image is collected by the vehicle camera, and the distant target region in the image is intercepted. Then, the true values ​​of various attributes of the distant target are manually annotated with the assistance of instruments, and the true values ​​are used as labels. The cropped region images and the corresponding labels are used as training samples to iterate the model, thereby obtaining the trained second detection network 22; the third detection network 23 is a neural network model specifically used for detecting nearby targets, and its training data source is as follows: Figure 4 As shown, Figure 4In the figure, region 41 is a near target region in the image, which indicates the region of the front vehicle that is close to the vehicle in the image. In implementation, the image is collected by the vehicle camera, the image is scaled, and the true values ​​of various attributes of the near target region are annotated in the scaled image. The scaled image and the annotated data are used as training samples to iterate the model, thereby obtaining the trained third detection network 23.

[0055] The workflow of this embodiment includes:

[0056] S201, determining N frames in the image sequence to be detected as a detection cycle; the value of N can be dynamically determined according to the actual perception frame rate of the vehicle-mounted camera;

[0057] S202, performing preliminary detection on the first frame image within the detection period through the first detection network 21 to obtain regional position information and confidence levels of all targets in the image;

[0058] S203, intercepting the distant target area based on the preliminary inspection result;

[0059] S204, clustering all intercepted distant target regions according to a clustering algorithm to divide the distant targets to be detected into K categories, where the value of K is N-1;

[0060] S205, according to the confidence of each distant target, calculate the cumulative confidence of each category, and arrange the divided K categories in ascending order according to the cumulative confidence, and then allocate the divided K categories to the second frame image to the Nth frame image in the detection cycle in sequence according to the arrangement order;

[0061] S206, according to the allocation result, cutting out a distant target area map from the second frame image to the Nth frame image within the detection cycle;

[0062] S207, performing target detection on the distant target area map through the second detection network 22 to obtain distant target information; compared with the traditional solution of processing all targets in each frame, the distant target detection method of the embodiment can effectively reduce the computing consumption while ensuring the accuracy and reliability of the detection results, and because a neural network model dedicated to detecting distant targets is used to perform the distant target detection task, the accuracy of perception is effectively improved;

[0063] S208, selecting a frame of image corresponding to an intermediate moment from the second frame of image to the Nth frame of image, and cutting out a near target area map from the selected image; wherein this step and S203 have no fixed order;

[0064] S209. Target detection is performed on the near target area map through the third detection network 23 to obtain near target information. Since the existence of near targets can easily significantly affect driving safety decisions, such as when facing pedestrians crossing or sudden stops of targets ahead, the vehicle needs to adopt corresponding processing measures in a timely manner. Therefore, in order to ensure stable and accurate detection of near targets, this embodiment of the invention selects the near target area corresponding to the image at the middle moment within the detection cycle and sends it to the neural network model specifically responsible for detecting near targets for inference and prediction. Compared with the traditional method of using one model to detect all targets, this embodiment of the invention can effectively improve the inference efficiency and inference accuracy.

[0065] The scheme of this embodiment can adaptively detect distant targets in batches by setting the detection cycle, effectively avoiding the waste of computing resources, and dynamically executing different tasks through three detection models. Under the premise of ensuring stable detection of distant and near targets, the perception performance is improved. Moreover, the traditional scheme only uses a large neural network model to complete the target detection task, which has high requirements on computing power. Since the scheme of this embodiment splits the task, the first detection network, the second detection network and the third detection network can all be made relatively small, and even on a chip platform with lower computing power, a better effect of detecting distant and near targets can be achieved.

[0066] Corresponding to the embodiments of the aforementioned method, the present application also provides embodiments of a target detection device and a terminal using the same:

[0067] like Figure 5 As shown, Figure 5 is a block diagram of a target detection device provided in an embodiment of the present application, the device comprising:

[0068] A preliminary detection module 51 is used to process the first frame image in the target image sequence collected by the vehicle-mounted camera through a first target detection model to obtain preliminary detection information; the preliminary detection information includes the regional position information of all targets in the first frame image;

[0069] A region cropping module 52 is used to obtain a distant target region according to the preliminary detection information, divide the distant target region into a plurality of categories, assign the plurality of categories to the remaining images in the target image sequence, crop a distant target region map corresponding to the assigned category from the remaining images, select a frame of image from the remaining images according to the timeline of the target image sequence, and crop a near target region map from the selected image;

[0070] The target detection module 53 is used to perform target detection on the distant target area map and the near target area map to obtain distant target information and near target information; the distant target information includes the position and posture information of the distant target in the target image sequence; the near target information includes the position and posture information of the near target in the target image sequence.

[0071] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0072] This application also provides an electronic device, see Figure 6 , Figure 6 A block diagram of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 610, a communication interface 620, a memory 630, and at least one communication bus 640. The communication bus 640 is used to realize direct connection and communication between these components. The communication interface 620 of the electronic device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 610 may be an integrated circuit chip with signal processing capabilities.

[0073] The processor 610 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 610 can also be any conventional processor, etc.

[0074] The memory 630 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 630 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 610, the electronic device may execute the above-mentioned Figure 1The method embodiment involves various steps.

[0075] Optionally, the electronic device may further include a storage controller and an input / output unit.

[0076] The memory 630, storage controller, processor 610, peripheral interface, input and output unit components are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 640. The processor 610 is used to execute executable modules stored in the memory 630, such as software function modules or computer programs included in the electronic device.

[0077] The input and output unit is used to provide users with the task creation and to create a start optional time period or preset execution time for the task to realize the interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.

[0078] Understandably, Figure 6 The structure shown is for illustration only, and the electronic device may also include Figure 6 More or fewer components as shown, or with Figure 6 Different configurations are shown. Figure 6 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0079] An embodiment of the present application further provides a storage medium having instructions stored thereon. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, the method will not be described here.

[0080] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0081] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0083] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0084] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0085] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0086] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A target detection method, characterized in that: include: Processing the first frame image in the target image sequence collected by the vehicle-mounted camera through the first target detection model to obtain preliminary detection information; The preliminary detection information includes regional position information of all targets in the first frame image; Acquire a distant target region according to the preliminary detection information, divide the distant target region into a plurality of categories, assign the plurality of categories to the remaining images in the target image sequence, crop a distant target region map corresponding to the assigned categories from the remaining images, select a frame of image from the remaining images according to a timeline of the target image sequence, and crop a near target region map from the selected image; Target detection is performed on the distant target area map and the near target area map to obtain distant target information and near target information; the distant target information includes the position and posture information of the distant targets in the target image sequence; the near target information includes the position and posture information of the near targets in the target image sequence.

2. The method according to claim 1, characterized in that The number of image frames of the target image sequence is set according to the perception frame rate of the vehicle-mounted camera.

3. The method according to claim 1, characterized in that The step of acquiring a distant target area according to the preliminary detection information and dividing the distant target area into a plurality of categories includes: According to the regional position information of all distant targets in the first frame of image, intercepting the distant target region in the first frame of image; A clustering algorithm is used to cluster the intercepted distant target areas to obtain multiple categories.

4. The method according to claim 3, characterized in that The clustering algorithm is a K-means clustering algorithm; the K value of the K-means clustering algorithm is a value obtained by subtracting one from the number of image frames of the target image sequence.

5. The method according to claim 4, characterized in that The preliminary detection information also includes the confidence of all targets in the first frame of image; The assigning the plurality of categories to the remaining images in the target image sequence comprises: Calculating the cumulative confidence of each category based on the confidence of the distant objects included in the category; Arrange the multiple categories in ascending order according to the cumulative confidence; According to the arrangement order of the multiple categories, the multiple categories are sequentially assigned to the remaining images in the target image sequence, so that each frame of the remaining images is assigned to one category.

6. The method according to claim 1, characterized in that The selecting a frame of image from the remaining images according to the time line of the target image sequence comprises: Determine, according to the timeline of the target image sequence, at least one frame of image corresponding to an intermediate moment in the remaining images; A frame of image is selected from at least one frame of image corresponding to the intermediate moment.

7. The method according to claim 1, characterized in that The performing target detection on the distant target area map and the near target area map to obtain distant target information and near target information includes: Performing target detection on the distant target area map by using a second target detection model to obtain distant target information, and performing target detection on the near target area map by using a third target detection model to obtain near target information; Among them, the second target detection model is trained based on the first training data set; the first training data set includes multiple distant target sample images and a label corresponding to each distant target sample image; the label corresponding to the distant target sample image includes the true value of the specified attribute of the distant target in the distant target sample image; the third target detection model is trained based on the second training data set; the second training data set includes multiple near target sample images and a label corresponding to each near target sample image; the label corresponding to the near target sample image includes the true value of the specified attribute of the near target in the near target sample image.

8. A target detection device, characterized in that: include: A preliminary detection module, used to process the first frame image in the target image sequence collected by the vehicle-mounted camera through a first target detection model to obtain preliminary detection information; The preliminary detection information includes regional position information of all targets in the first frame image; A region cropping module, used for acquiring a distant target region according to the preliminary detection information, dividing the distant target region into a plurality of categories, assigning the plurality of categories to the remaining images in the target image sequence, cropping a distant target image corresponding to the assigned category from the remaining images, selecting a frame of image from the remaining images according to a timeline of the target image sequence, and cropping a near target region image from the selected image; The target detection module is used to perform target detection on the distant target area map and the near target area map to obtain distant target information and near target information; the distant target information includes the position and posture information of the distant targets in the target image sequence; the near target information includes the position and posture information of the near targets in the target image sequence.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images