Target detection method and device

Abstract

An embodiment of the present application provides a target detection method and device, which includes: acquiring an image using a camera device; calibrating a region of interest in the image based on parameters of the camera device and a preset driving path; detecting the image using a target detection algorithm to obtain the category to which the target object in the image belongs, a first position area of the target object in the image, and a confidence level of the category to which the target object belongs; and correcting the confidence level of the category to which the target object belongs based on the relative positional relationship between the first position area and the region of interest to obtain a first confidence level, thereby making the detected target object more accurate.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a target detection method and device. Background Art

[0002] With the advancement of science and technology, artificial intelligence (AI) technology has made rapid progress. In some AI technologies, machine learning methods are often used to construct initial models of various structures, such as neural network models, support vector machine models, and decision tree models. These initial models are then trained to achieve objectives such as image recognition and natural language processing. Image recognition also includes recognizing text and detecting objects in images.

[0003] Related object detection technologies often fail to effectively identify distant or small objects on the road in scenarios requiring small object detection, such as autonomous driving. For example, a pedestrian sign on a distant road sign might be mistaken for a pedestrian on the road.

[0004] Therefore, how to effectively identify small targets in images becomes a problem that needs to be solved. Summary of the Invention

[0005] By adopting the target detection method and device shown in this application, the probability of misidentifying small targets in an image can be reduced, which is conducive to improving the detection accuracy of target detection.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a target detection method, which is applied to an electronic device. The target detection method includes: acquiring an image using a camera device; calibrating a region of interest in the image based on parameters of the camera device and a preset driving path; detecting the image using a target detection algorithm to obtain the category to which the target object in the image belongs, a first position area of the target object in the image, and a confidence level of the category to which the target object belongs; and correcting the confidence level of the category to which the target object belongs based on the relative position relationship between the first position area and the region of interest to obtain a first confidence level.

[0008] By using the positional relationship between the location area where the detected target object is located and the area of interest to determine the first confidence of the category to which it belongs, it is possible to further verify whether the category to which the detected target object belongs is correct, and filter out some illogical or unreasonable category detection results, thereby improving the accuracy of target detection.

[0009] Based on the first aspect, when the first confidence level is less than a preset threshold, the first confidence level can be directly output; when the first confidence level is less than the preset threshold, the target detection method further includes: determining the second position area in the image based on the parameters of the camera device, the boundary coordinates of the first position area in the image, and the size of the object corresponding to the preset category in the real world; detecting the error between the first position area and the second position area; and based on the error, correcting the first confidence level to obtain the second confidence level of the category to which the target belongs.

[0010] By using the error between the second position area and the first position area to correct the first confidence level, the misjudgment of target objects on the road (for example, misjudging trees far away on the road as pedestrians) can be further reduced, thereby further improving the accuracy of target object detection.

[0011] In one possible implementation, the parameters of the camera device include at least one of the following: the focal length of the camera device, the distance between the camera device and the reference surface, the conversion matrix from the camera device coordinate system to the image coordinate system, and the size of the photosensitive unit in the photosensitive element.

[0012] In one possible implementation, the determining of the second position area in the image based on the parameters of the camera device, the boundary coordinates of the first position area in the image, and the size of the object corresponding to the category in the real world includes: determining the distance between the camera device and the target object based on the focal length of the camera device, the distance between the camera device and the reference surface, the conversion matrix from the camera device coordinate system to the image coordinate system, the size of the photosensitive unit in the photosensitive element, and the boundary coordinates of the first position area in the image; determining the second position area in the image based on the distance between the camera device and the target object, the size of the object corresponding to the detected category in the real world, the distance between the camera device and the reference surface, and the boundary coordinates of the first position area.

[0013] In a possible implementation, the category to which the target object belongs is selected from the preset candidate categories based on matching results of features of the target object with features of objects corresponding to multiple preset candidate categories.

[0014] Based on the first aspect, in a possible implementation, the target detection algorithm may be a pre-trained target detection model. The method of detecting the image using the target detection algorithm to obtain the category to which the target object in the image belongs, the first position area of the target object in the image, and the confidence level of the category to which the target object belongs comprises: setting calibration parameters in the pre-trained target detection model, the calibration parameters being used to instruct the target detection model to calibrate multiple candidate areas in the image; inputting the image into the target detection model to obtain the output result of the target detection model, the output result being used to indicate whether each candidate area presents an object of a preset candidate category and the confidence level of the category to which the target object belongs, wherein the target detection model is obtained by training a neural network based on training samples and calibration parameters for calibrating candidate areas.

[0015] Based on the first aspect, in a possible implementation method, the multiple candidate areas in the image are predetermined based on constraints; the constraints include: the area range in which the objects corresponding to each of the preset candidate categories appear in the image, and the imaging size range in the image of the objects corresponding to each of the preset candidate categories.

[0016] By using constraint conditions to filter the calibrated position areas, some position areas that are unnecessary to be detected can be filtered out, reducing the number of position areas to be detected in the image, thereby improving the detection speed and detection accuracy of the target detection model.

[0017] In a possible implementation, determining the multiple candidate regions in the image includes: calibrating an initial candidate region in the image; screening the initial candidate region using the constraint condition, and obtaining the multiple candidate regions based on the screening result.

[0018] In one possible implementation, the method also includes an optimization step for the target detection model, and the optimization step includes: obtaining a training sample set, the training sample set including multiple sample images, each of the sample images presenting a target object; inputting the sample image into the target detection model to obtain the category to which the target object in each sample image belongs and the first position area of the target object in the sample image, and determining the second position area in each sample image based on the category to which the target object in the sample image belongs, the boundary coordinates of the first position area, and the parameters of the shooting device used to shoot the sample image; using a preset loss function to determine the deviation between the first position area and the second position area in each training sample, and based on the deviation, iteratively adjusting the target detection model to obtain an optimized target detection model.

[0019] By optimizing the target detection model, the detection accuracy of the target detection model can be further improved, that is, the accuracy of road target detection can be improved, providing guarantees for subsequent autonomous driving vehicles to detect and avoid obstacles.

[0020] In a second aspect, an embodiment of the present application provides a target detection device, which includes: an acquisition module for acquiring an image using a camera device; a calibration module for calibrating a region of interest in the image based on parameters of the camera device and a preset driving path; a first detection module for detecting the image using a target detection algorithm to obtain the category to which the target object in the image belongs, a first position area of the target object in the image, and a confidence level of the category to which the target object belongs; and a first correction module for correcting the confidence level of the category to which the target object belongs based on the relative positional relationship between the first position area and the region of interest to obtain a first confidence level.

[0021] Based on the second aspect, in a possible implementation, the target detection device also includes: a determination module for determining, in response to the first confidence being greater than a preset threshold, a second position area in the image based on parameters of the camera device, boundary coordinates of the first position area in the image, and a preset size of the object corresponding to the category in the real world; a second detection module for detecting an error between the first position area and the second position area; and a second correction module for correcting the first confidence based on the error to obtain a second confidence of the category to which the target belongs.

[0022] Based on the second aspect, in a possible implementation, the parameters of the camera device include at least one of the following: the focal length of the camera device, the distance between the camera device and the reference surface, the conversion matrix from the camera device coordinate system to the image coordinate system, and the size of the photosensitive unit in the photosensitive element.

[0023] Based on the second aspect, in a possible implementation, the determination module includes: a first determination submodule, used to determine the distance between the camera device and the target object based on the focal length of the camera device, the distance between the camera device and the reference surface, the conversion matrix from the camera device coordinate system to the image coordinate system, the size of the photosensitive unit in the photosensitive element, and the boundary coordinates of the first position area in the image; a second determination submodule, used to determine the second position area in the image based on the distance between the camera device and the target object, the size of the object corresponding to the detected category in the real world, the distance between the camera device and the reference surface, and the boundary coordinates of the first position area.

[0024] Based on the second aspect, in a possible implementation, the category to which the target object belongs is selected from the preset candidate categories based on matching results of the feature of the target object with features of objects corresponding to multiple preset candidate categories.

[0025] Based on the second aspect, in a possible implementation, the first detection module includes: a setting submodule for setting calibration parameters in a pre-trained target detection model, and the calibration parameters are used to instruct the target detection model to calibrate multiple candidate areas in the image; a detection submodule for inputting the image into the target detection model to obtain the output result of the target detection model, and the output result is used to indicate whether an object of a preset candidate category and the confidence of the category to which the target object belongs are presented in each of the candidate areas, wherein the target detection model is obtained by training a neural network based on training samples and calibration parameters for candidate area calibration.

[0026] Based on the second aspect, in a possible implementation method, the multiple candidate areas in the image are predetermined based on constraints; the constraints include: the area range in which the objects corresponding to each of the preset candidate categories appear in the image, and the imaging size range in which the objects corresponding to each of the preset candidate categories appear in the image.

[0027] Based on the second aspect, in a possible implementation, the setting submodule is specifically used to: mark an initial candidate area in the image; filter the initial candidate area using the constraint conditions, and obtain the multiple candidate areas based on the filtering results.

[0028] Based on the second aspect, in a possible implementation method, the target detection device also includes a model optimization module, and the model optimization module is specifically used to: obtain a training sample set, the training sample set includes multiple sample images, and each of the sample images presents a target object; input the sample image into the target detection model to obtain the category of the target object in each sample image and the first position area of the target object in the sample image, and determine the second position area in each sample image based on the category of the target object in the sample image, the boundary coordinates of the first position area, and the parameters of the shooting device used to shoot the sample image; use a preset loss function to determine the deviation between the first position area and the second position area in each training sample, and based on the deviation, iteratively adjust the target detection model to obtain an optimized target detection model.

[0029] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method described in the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are run on a computer, they are used to execute the method described in the first aspect above.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program or a computer program product. When the computer program or the computer program product is executed on a computer, it enables the computer to execute the method described in the first aspect.

[0032] It should be understood that the second to fifth aspects of this application are consistent with the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar, so they will not be repeated. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the hardware structure of an application scenario provided by an embodiment of the present application;

[0034] Figure 2 This is a schematic diagram of the prior art of misjudging an object presented in an image provided by an embodiment of the present application;

[0035] Figure 3 is a schematic flow chart of a target detection method provided in an embodiment of the present application;

[0036] Figure 4 This is a schematic diagram of negative samples provided during the training of the target detection model provided in an embodiment of the present application;

[0037] Figure 5a-5e This is a schematic diagram of an application scenario of the target detection method provided in an embodiment of the present application;

[0038] Figure 6 is a schematic flow chart of another target detection method provided in an embodiment of the present application;

[0039] Figure 7 This is another schematic diagram of an application scenario of the target detection method provided in an embodiment of the present application;

[0040] Figure 8a-Figure 8b The embodiment of this application provides Figure 7 A schematic diagram of a method for determining a second location area in the application scenario shown;

[0041] Figure 9 Schematic diagram of a target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. In the following description, reference is made to the drawings that form a part of the present application and illustrate specific aspects of the embodiments of the present application or specific aspects of the embodiments of the present application that can be used. It should be understood that the embodiments of the present application can be used in other aspects and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description should not be understood in a restrictive sense, and the scope of the present application is defined by the appended claims. For example, it should be understood that the disclosure in conjunction with the described method can also be applied to the corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device can include one or more units such as functional units to perform the one or more method steps described (for example, one unit performs one or more steps, or multiple units, each of which performs one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific device is described based on one or more units, such as functional units, the corresponding method may include a step to perform the functionality of the one or more units (e.g., a step to perform the functionality of the one or more units, or multiple steps, each of which performs the functionality of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Further, it should be understood that unless otherwise explicitly stated, the features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0043] The object detection method described in this application can be applied to various scenarios in the field of image recognition, where small objects in images need to be detected and identified. The following uses the detection of small objects on the road in an autonomous driving scenario as an example to explain this application in detail.

[0044] Please refer to Figure 1 , Figure 1 Schematic diagram of the structure of the vehicle in the embodiment of the present application.

[0045] Components coupled to or included in vehicle 100 may include a propulsion system 110, a sensor system 120, a control system 130, peripheral devices 140, a power source 101, a computing device 107, and a user interface 108. Computing device 107 includes a processor 102, a transceiver 103, and a memory 104. Computing device 107 may be a controller or a portion of a controller for vehicle 100. Memory 104 includes instructions 106 that processor 102 can execute and may also store map data 105. The components of vehicle 100 may be configured to operate in a manner interconnected with each other and / or with other components coupled to various systems. For example, power source 101 may provide power to all components of vehicle 100. Computing device 107 may be configured to receive data from and control propulsion system 110, sensor system 120, control system 130, and peripheral devices 140. Computing device 107 may be configured to generate images for display on user interface 108 and to receive input from user interface 108.

[0046] In some possible embodiments, the vehicle 100 may further include more, fewer, or different systems, and each system may include more, fewer, or different components. In addition, the systems and components shown may be combined or divided in any manner, and the embodiments of the present application do not specifically limit this.

[0047] Next, each of the above systems will be described.

[0048] The propulsion system 102 can be used to provide powered movement for the vehicle 100. Figure 1 As shown, the propulsion system 102 may include an engine / motor 114, an energy source 113, a transmission 112, and wheels / tires 111. Of course, the propulsion system 102 may also additionally or alternatively include Figure 1 The embodiments of the present application do not specifically limit other components besides the components shown.

[0049] The sensor system 104 may include several sensors for sensing information about the environment in which the vehicle 100 is located. As shown, the sensors of the sensor system include at least one of a global positioning system (GPS) 126, an inertial measurement unit (IMU) 125, a lidar sensor 124, a visual sensor 123, a millimeter-wave radar sensor 122, and an actuator 121 for modifying the position and / or orientation of the sensor. The sensor system 120 may also include additional sensors, including, for example, sensors that monitor the internal systems of the vehicle 100 (e.g., at least one of an O2 monitor, a fuel gauge, and an oil temperature). The sensor system 120 may also include other sensors.

[0050] Global Positioning System (GPS) module 126 may be any sensor for estimating the geographic location of vehicle 100. To this end, GPS module 126 may include a transceiver that estimates the position of vehicle 100 relative to the Earth based on satellite positioning data. In an example, computing device 107 may be configured to use GPS module 126 in conjunction with map data 105 to estimate the location of lane boundaries on a road on which vehicle 100 may be traveling. GPS module 126 may also take other forms.

[0051] IMU 125 can be used to sense the position and orientation changes of vehicle 100 based on inertial acceleration and any combination thereof. In some examples, the combination of sensors can include, for example, an accelerometer and a gyroscope. Other combinations of sensors are also possible.

[0052] The LiDAR (Light Detection and Ranging) sensor 124 can be considered an object detection system that uses light to sense or detect objects in the environment in which the vehicle 100 is located. Generally, the LIDAR 124 is an optical remote sensing technology that can measure the distance to a target or other properties of a target by illuminating the target with light. As an example, the LIDAR 124 may include a laser source and / or a laser scanner configured to emit laser pulses, and a detector for receiving reflections of the laser pulses. For example, the LIDAR 124 may include a laser rangefinder reflected by a rotating mirror and scan the laser in one or two dimensions around a digitized scene to collect distance measurements at specified angular intervals. In an example, the LIDAR 124 may include components such as a light source (e.g., a laser), a scanner and optical system, a light detector and receiver electronics, and a position and navigation system. The LIDAR 124 determines the distance to an object by scanning the laser reflected from the object, and can form a three-dimensional (3D) image of the environment with an accuracy of up to centimeters.

[0053] The visual sensor 123 can be any camera (e.g., a still camera, a video camera, etc.) that can be used to acquire images of the environment in which the vehicle 100 is located. To this end, the visual sensor 123 can be configured to detect visible light, or it can be configured to detect light from other parts of the spectrum (e.g., infrared light or ultraviolet light). Other types of visual sensors are also possible. The visual sensor 123 can be a two-dimensional detector, or a detector that can have a three-dimensional spatial range. In some possible embodiments, the visual sensor 123 can be, for example, a distance detector configured to generate a two-dimensional image indicating the distance from the visual sensor 123 to several points in the environment. To this end, the visual sensor 123 can utilize one or more distance detection techniques. For example, the visual sensor 123 can be configured to utilize structured light technology, in which the vehicle 100 illuminates an object in the environment with a predetermined light pattern, such as a grid or checkerboard pattern, and uses the visual sensor 123 to detect reflections of the predetermined light pattern from the object. Based on distortions in the reflected light pattern, the vehicle 100 can be configured to detect the distance to a point on the object. The predetermined light pattern can include infrared light or light of other wavelengths.

[0054] Millimeter-wave radar sensors (122) typically refer to object detection sensors with a wavelength of 1 to 10 mm and a frequency range of approximately 10 GHz to 200 GHz. Millimeter-wave radar measurements provide depth information and can provide the distance to a target. Furthermore, due to the significant Doppler effect, millimeter-wave radar is highly sensitive to velocity and can directly determine the target's velocity. This velocity can be extracted by detecting the Doppler shift. Currently, the two mainstream automotive millimeter-wave radar frequency bands are 24 GHz and 77 GHz. The former, with a wavelength of approximately 1.25 cm, is primarily used for short-range sensing, such as vehicle surroundings, blind spots, parking assistance, and lane change assistance. The latter, with a wavelength of approximately 4 mm, is used for medium- and long-range sensing, such as automatic following, adaptive cruise control (ACC), and autonomous emergency braking (AEB).

[0055] The control system 130 may be configured to control the operation of the vehicle 100 and its components. To this end, the control system 130 may include a steering unit 136, a throttle 135, a brake unit 134, a sensor fusion unit 133, a computer vision system 132, and a navigation or path control system 131. Of course, the control system 130 may also additionally or alternatively include other components besides Figure 1 The embodiments of the present application do not specifically limit other components besides the components shown.

[0056] The peripheral devices 140 may be configured to allow the vehicle 100 to interact with external sensors, other vehicles, and / or users. To this end, the peripheral devices 140 may include, for example, a wireless communication system 144, a touch screen 143, a microphone 142, and / or a speaker 141. Of course, the peripheral devices 140 may additionally or alternatively include other than Figure 1 The embodiments of the present application do not specifically limit other components besides the components shown.

[0057] Power source 101 can be configured to provide power to some or all components of vehicle 100. To this end, power source 110 can include, for example, a rechargeable lithium-ion or lead-acid battery. In some examples, one or more battery packs can be configured to provide power. Other power source materials and configurations are also possible. In some possible implementations, power source 110 and energy source 113 can be implemented together.

[0058] The processor 102 included in the computing device 107 may include one or more general-purpose processors and / or one or more specialized processors (e.g., an image processor, a digital signal processor, etc.). To the extent that the processor 102 includes more than one processor, the processors may operate individually or in combination. The computing device 107 may implement functions for controlling the vehicle 100 based on input received through the user interface 108.

[0059] The transceiver 103 is used for communication between the computing device 107 and various systems.

[0060] The memory 104 may further include one or more volatile storage components and / or one or more non-volatile storage components, such as optical, magnetic, and / or organic storage devices, and the memory 104 may be fully or partially integrated with the processor 102. The memory 104 may contain instructions 106 (e.g., program logic) that can be executed by the processor 102 to perform various vehicle functions, including any of the functions or methods described in the embodiments of the present application.

[0061] The components of the vehicle 100 can be configured to work in an interconnected manner with other components within and / or outside of their respective systems. To this end, the components and systems of the vehicle 100 can be connected together through a system bus, a network and / or other connection mechanisms.

[0062] In the embodiments of the present application, combined with the structure of the vehicle 100, the vehicle typically employs an object detection algorithm to detect objects on the road in real time during autonomous driving mode to ensure vehicle safety. For example, object detection can inform the vehicle of drivable areas and identify the locations of obstacles, thereby assisting the vehicle in avoiding them.

[0063] When using an object detection algorithm, a computing device first uses deep learning to train a neural network capable of identifying specific object categories. These categories can include common objects such as pedestrians, vehicles, trees, houses, and road infrastructure. During object detection, the computing device uses this neural network to identify these specific object categories. Because the neural network learns the characteristics of each specific object category, it often fails to effectively identify similar features in an image, leading to misjudgments.

[0064] For example Figure 2 As shown, in Figure 2 In the example, a pedestrian icon is displayed on the sign, but the pedestrian is not an actual pedestrian. However, the characteristics of the pedestrian icon on the sign are usually similar to those of pedestrians far away on the road, causing the neural network to mistakenly identify the pedestrian icon on the sign as a small pedestrian on the road, for example Figure 2As shown, the neural network judges Figure 2 The probability that the pedestrian on the sign is a pedestrian is 0.85, which reduces the accuracy of object detection.

[0065] To solve the above problems, an embodiment of the present application provides a target detection method, which can be applied to a target detection device. The target detection device can be the computing device described in the above embodiments or a part of the computing device.

[0066] Please refer to Figure 3 , Figure 3 This is a schematic flow chart of the target detection method shown in the embodiment of this application. Figure 3 As shown, the method includes:

[0067] S301, using a camera to acquire an image.

[0068] The camera device here is the visual sensor in the aforementioned sensor system, which is used to capture images of the road ahead of the vehicle. This image can include objects such as pedestrians, vehicles, road surfaces, barriers, and of course, sidewalks, roadside trees, and traffic lights, although this embodiment of the application does not specifically limit this.

[0069] In practical applications, the camera device can be a monocular camera that captures an image to be processed at a given moment. Alternatively, the camera device can include multiple cameras, which can be physically combined in one camera device or physically separated into multiple camera devices. Multiple cameras can capture multiple images at the same time, and these images can be processed to obtain an image to be recognized. Of course, the camera device can also be in other situations, and the embodiments of the present application do not specifically limit this.

[0070] In a specific implementation, the camera device can capture images in real time, or can capture images periodically. The period may be 3s, 5s, 10s, etc. The camera device can also capture images in other ways, which are not specifically limited in the embodiments of the present application. After the camera device captures the image, it can pass the image to the above-mentioned target detection device, at which point the target detection device can obtain the image. It should be noted that S301 can be executed after the vehicle is started, or after the vehicle starts the automatic driving function.

[0071] In step S302 , a region of interest (ROI) in the image is calibrated based on the parameters of the camera device and the preset driving path.

[0072] The image captured by the above-mentioned camera device is usually a road condition image. The image usually shows objects in front of and on both sides of the road. For example, vehicles in front of or on the side of the road, pedestrians, trees on both sides of the road, tires in the lane, wooden boxes, etc. Therefore, the region of interest in the image can be the road in front and the roads on both sides. In the image, the ROI can be an area outlined in the form of a box, circle, ellipse or irregular polygon. For example Figure 5b As shown, it schematically shows the ROI in the image.

[0073] In a specific implementation, the ROI can be determined based on the camera parameters and the preset driving path. Camera parameters include, but are not limited to, the height between the camera and the reference plane and the location of the camera on the vehicle. The preset driving path includes, but is not limited to, the vehicle's lane and the vehicle's driving rules (e.g., keeping left or keeping right). After acquiring an image, the target detection device can demarcate the region of interest in the image based on the camera parameters and the preset driving path.

[0074] Step S303 : Detect the acquired image using a target detection algorithm to obtain the category to which the target object in the image belongs, the location area of the target object in the image, and the confidence level of the category to which the target object belongs.

[0075] Step S303 can be specifically implemented through the following steps:

[0076] Step 1: Set calibration parameters in the target detection model. The calibration parameters are used to instruct the target detection model to calibrate multiple candidate regions in the image.

[0077] Here, the calibration parameters are the position parameters (e.g., position coordinates) of each candidate region in the image. The position of the candidate region in the image is predetermined. Determining the candidate region here can also be called setting a prior anchor.

[0078] In practice, various existing methods such as manual setting, K-means clustering, RPN (region proposal network) algorithm or selective search algorithm can be used to determine multiple candidate regions. Then, the position parameters of each candidate region in the image are set in the object detection model. The method of determining candidate regions can be implemented in the following two ways:

[0079] As a possible implementation, the image is divided into multiple equal units based on its size. Then, for each equally divided unit in the image, multiple prior frames with different aspect ratios are set. The aspect ratio differences of these multiple prior frames can be in an arithmetic progression. These multiple prior frames are also multiple candidate regions.

[0080] As another possible implementation, a segmentation method is used to segment the image to obtain a set of candidate region images representing multiple image blocks. The similarity between each two adjacent regions in the candidate region image set is calculated (e.g., this can be determined based on multiple dimensions such as color similarity or texture similarity). Finally, regions with similarities exceeding a preset threshold are merged to determine the final candidate regions.

[0081] Step 2: Input the image into a pre-trained, calibrated object detection model so that the object detection model detects the confidence level of each candidate region representing a target object of a preset candidate category and the candidate region's positioning deviation. When the confidence level of one or more candidate regions representing one or more objects of the preset candidate category is high and exceeds a threshold, it is predicted that the region or regions represent an object of the preset candidate category. The positioning deviation of the candidate region is determined by the object detection model through edge detection and fully connected layer (or fully convolutional layer) detection of the candidate regions. The object detection model can adjust the position of the candidate region in the image based on the positioning deviation of the candidate region. The object detection model then outputs the position information of the candidate region representing the target object in the image, the category to which the target object represented in the candidate region belongs, and the confidence level of the category to which the target object belongs. The preset candidate category can include obstacles on the road, including but not limited to pedestrians, vehicles, trees, tires, boxes, construction signs, etc. The position information of the candidate region representing the target object in the image output by the object detection model is the position information after the candidate region's positioning deviation has been re-adjusted.

[0082] In a possible implementation, the candidate region described in the first step may also be determined based on constraint conditions.

[0083] In a specific implementation, the aforementioned selective search algorithm, manual setting, RPN or K-means clustering methods can be used to demarcate the initial candidate regions in the image; then, the initial candidate regions are screened using the constraints to obtain the final candidate regions.

[0084] The constraint conditions specifically refer to: the area range of each object in the preset candidate category presented in the image, and the imaging size range of each object in the preset candidate category in the image.

[0085] The range of the area within the image where each object in the preset candidate category appears is determined based on the possible real-world location of each object. In practice, the possible real-world location of each object can be mapped to the image based on the focal length of the camera, the size of the photosensitive cells in the photosensitive element, the optical center parameters of the camera, the position coordinates of the road surface in the world coordinate system, and the height between the camera and the reference surface. The mapped area in the image is the area within which the object appears.

[0086] The imaging size range of each object in the preset candidate category in the image is determined based on the distance between the target object and the camera device, and the height and width of each object in the real world. At different distances, the imaging size of objects of the same category in images taken by the same shooting device is different. In practice, the size of each object in the image at different distances can be determined based on the focal length of the camera device, the size of the photosensitive unit in the photosensitive element, the optical center parameters of the camera device, the position coordinates of the road surface in the world coordinate system, the height between the camera device and the reference surface, and the height and width of each object in the real world, and then the imaging size range of each object in the image can be determined.

[0087] As an example, assuming the candidate object category only includes pedestrians, the constraints include: the possible location of pedestrians is the road surface area, and the size of the pedestrian image in the image ranges from a to b, where a is smaller than b. Specifically, when the pedestrian is 50 meters from the camera, the image size in the image is a, and when the pedestrian is 300 meters from the camera, the image size in the image is b. 50 meters to 300 meters can be considered the range of distances that the camera can capture. The size of the pedestrian image in the image is determined based on the pedestrian's height and width in the real world, the distance from the camera, the camera's focal length, the size of the photosensitive cells in the photosensitive element, the optical center parameter of the camera, and the height between the camera and the reference surface. Based on the camera's focal length, optical center parameter, the road surface's position in the world coordinate system, and the height between the camera and the reference surface, the area of the road surface represented in the image can be calibrated, denoted as the road surface area. Then, candidate areas within the road surface area and with a size between a and b are selected from the initial candidate areas. The screened areas are then used as the final candidate areas.

[0088] The reference plane mentioned above may be a horizontal plane.

[0089] By using constraint conditions to filter the calibrated position areas, some position areas that are unnecessary to be detected can be filtered out, reducing the number of position areas to be detected in the image, thereby improving the detection speed and detection accuracy of the target detection model.

[0090] The target detection model described in the second step is obtained by training the neural network based on the training sample set and the preset loss function.

[0091] Specifically, the training sample set includes sample images and annotation information for the sample images. The target objects in the sample images are small road target objects, that is, the target objects occupy a small position in the image (eg, smaller than a preset area threshold).

[0092] Sample images include positive sample images and negative sample images. Negative sample images include images in which the presented object has the shape or outline of one of the preset candidate categories, but belongs to other categories. The other category may belong to another of the preset candidate categories, or may not belong to any of the preset candidate categories. For example, an image of a tree with a pedestrian outline in a local position, an image of a trash can with a puppy outline, or an image of a Figure 4 The image of the sign showing pedestrians is shown. The annotation information is used to indicate the category to which the object in the positive sample image belongs and the position of the object in the sample image. The annotation information is also used to indicate the category to which the object presented in the negative sample image belongs or does not belong to any preset candidate category, as well as the position of the object presented in the image in the sample image. As an example, assume that the preset candidate categories include two categories: pedestrians and trees. Among them, pedestrians are represented by 1, trees are represented by 2, and those that do not belong to any category are represented by 0. For example Figure 4 The negative sample’s label information is 0, and the location area is (a, b, c, d). a, b, c, d are the four vertices of the rectangular box shown in Figure 4. Figure 4 The image coordinates in .

[0093] Based on the size of the sample image and the size of the target to be detected, combined with the above constraints and the candidate region determination method, the candidate region is obtained, and then the position parameters of the candidate region in the image are set in the neural network.

[0094] The sample image is input into a neural network with parameter settings to obtain random output information, which includes the category corresponding to the object presented in the sample image, the location area in the image, and the confidence level of the category corresponding to the presented object.

[0095] Then, a preset loss function is used to calculate the deviation between the output information and the labeled information. Based on this deviation, the weight parameters of the neural network are iteratively adjusted to obtain the above-mentioned object detection model. The preset loss function here may include, but is not limited to, a mean square error function. The above-mentioned labeled information may include the confidence level of the annotation. The deviation between the calculated output information and the labeled information here may refer to the deviation between the confidence level of the category corresponding to the presented object and the confidence level of the annotation.

[0096] It should be noted that the above neural network may include convolutional layers, pooling layers, hidden layers, fully connected layers, etc. The number of layers can be determined according to the number of categories to be identified, the number of target categories presented in each image, and the number of pixels in the image.

[0097] Step S304 : Based on the determined relative positional relationship between the position area and the region of interest, the confidence level of the category to which the target object belongs is corrected to obtain a first confidence level.

[0098] Based on the boundary coordinates of the region of interest determined in step S303 and the boundary coordinates of the location region determined in step S302, it is determined whether the lower boundary coordinates of the location region are within the ROI. When the lower boundary of the first location is not within the ROI, the first confidence level can be set to a lower confidence level, such as 0 or 0.1. In other words, the probability that the target object belongs to this category is very low. When the lower boundary of the first location is within the ROI, the first confidence level can be set to a higher confidence level, such as 0.8, 0.9, or 1.

[0099] For example, when the detected target object is a pedestrian, they typically have contact with the ground, either with their feet or by traveling on a vehicle such as a motorcycle or bicycle. In this case, the ROI in the image is the ground. Next, the lower boundary of the region where the human image is located is determined to be within the ground, thereby determining whether the person depicted in the image is in contact with the ground. If the lower boundary of the region where the human image is located is within the ground, it indicates that the person is in contact with the ground, meaning that the confidence level is high that the target object is a person. Therefore, a higher confidence level is set for the first confidence level. If the lower boundary of the region where the human image is located is not within the ground, it indicates that the person is not in contact with the ground and is essentially floating in mid-air. Therefore, the confidence level is low that the target object is a person. Therefore, a lower confidence level is set for the first confidence level.

[0100] from Figure 3 It can be seen from the illustrated embodiment that by utilizing the positional relationship between the location area where the detected target object is located and the ROI to determine the first confidence level of the category to which it belongs, it is possible to further verify whether the category to which the detected target object belongs is correct, and filter out some illogical or unreasonable category detection results, thereby improving the accuracy of target detection.

[0101] Combine Figure 5a-5e The application scenario shown is Figure 3 The implementation of the target detection method is described in detail.

[0102] First, the camera device obtains Figure 5a Image A shown.

[0103] Next, the target detection device can determine the region of interest in image A. The region of interest in image A can be a ground range area, such as Figure 5b At this time, the target detection device set in the autonomous driving vehicle can record the boundary coordinates of the area of interest in image A.

[0104] Then, using the candidate region calibration method described in the first step above, multiple candidate region images are calibrated in the image to obtain image B, as shown in Figure 5c As shown. Figure 5c It can be seen from the figure that there are a large number of rectangular boxes distributed on the image B, where each rectangular box is a candidate region. Figure 5c The rectangular frames shown are for illustration only. In actual applications, more or fewer rectangular frames may be included, and the size of each rectangular frame may also be determined according to the requirements of the application scenario.

[0105] Then, the image B is input into the pre-trained target detection model to determine whether there is a target object of the preset candidate category in each candidate area, such as Figure 5d shown. Figure 5d The detection results output by the target detection model are shown. Figure 5d As can be seen from the figure, the target detection model detects that the target object presented by candidate region a is a pedestrian with a probability of 0.85; the target object presented by candidate region b is a tree with a probability of 0.9; and the target object presented by candidate region c is a pedestrian with a probability of 0.7. Candidate regions a, b, and c are the location areas detected by the target detection model.

[0106] Finally, the target detection device can compare the image coordinates of each candidate region image with the boundary coordinates of the region of interest to determine whether the image coordinates of each candidate region image are within the range of the region of interest. Figure 5e As shown, candidate region c is not within the image's region of interest. A pedestrian's feet should be in contact with the ground, but the pedestrian's feet in candidate region c are not. Therefore, the object detection device can determine that the object in candidate region image c is a pedestrian with a confidence level of 0.1.

[0107] Please continue to refer to Figure 6 , which shows a flow chart of another embodiment of the target detection method provided by the present application, the target detection method includes:

[0108] S601: Acquire an image using a camera device.

[0109] S602: Based on the parameters of the camera device and the preset driving path, a region of interest in the image is calibrated.

[0110] S603 , detecting the acquired image using a target detection algorithm to obtain the category to which the target object in the image belongs, the location area of the target object in the image, and the confidence level of the category to which the target object belongs.

[0111] S604: Based on the determined relative position relationship between the position area and the region of interest, correct the confidence level of the category to which the target object belongs to obtain a first confidence level.

[0112] The specific implementation of steps S601 to S604 and the beneficial effects thereof are shown in FIG. Figure 3 The description of steps S301 to S304 in the illustrated embodiment will not be repeated here.

[0113] S605: Check whether the first confidence level of the category of the target object determined in step S604 is greater than a preset threshold. If it is greater than the preset threshold, execute steps S606 to S608. If it is less than or equal to the preset threshold, output the value of the first confidence level.

[0114] S606 , determining a second position area in the image based on the parameters of the camera device, the boundary coordinates of the first position area in the image, and the size of the object corresponding to the preset category in the real world.

[0115] The parameters of the camera device here specifically include but are not limited to: the focal length of the camera device, the size of the photosensitive unit in the photosensitive element, the conversion matrix of the camera device, the distance between the camera device and the reference surface, and the optical center parameters of the camera device.

[0116] The reference surface here can be the ground.

[0117] Step S606 can be specifically implemented through the following steps.

[0118] Step 1: Determine the distance between the camera device and the target object based on the focal length of the camera device, the distance between the camera device and the reference surface, the conversion matrix from the camera device coordinate system to the image coordinate system, the size of the photosensitive unit in the photosensitive element, and the boundary coordinates of the first position area in the image.

[0119] Specifically, assuming the target object detected in step S604 belongs to the correct category, the size of the object corresponding to that category in the real world is queried. For example, the height of pedestrians on the road is generally between 130cm and 190cm, and the horizontal width is generally between 43cm and 55cm. Using formula (1), which is the conversion formula between the world coordinate system and the image plane coordinate system, the distance between the camera device and the target object can be determined.

[0120] Z w p=KR[I|-T]P w (1)

[0121] Among them, P w =[X w Y w Z w 1] T is the coordinate of the target in the world coordinate system, p = [uv 1] T is the imaging coordinate of the target in the image, K is the transformation matrix of the camera device, R is the rotation matrix from the world coordinate system to the camera device coordinate system, T is the translation matrix from the world coordinate system to the camera device coordinate system, where the height of the camera device is set in the matrix, I is the unit diagonal matrix, Z w Represents the distance between the camera and the target object.

[0122] In addition, formula (2) is a formula for determining the distance between the camera device and the target object obtained by refining and deducing formula (1) and applied in the embodiments of the present application. Formula (2) is used to determine the distance Zw between the camera device and the target object, i.e., based on the size of the target in the real world and the size of the target in the imaging plane.

[0123] Z w =h w f y / (hd y )

[0124] or Z w =w w f x / (wd x ) (2)

[0125] Among them, f x ,f y are the focal lengths of the camera in the x and y axes, respectively, and d x ,d y They are the sizes of the photosensitive units in the x and y axis directions on the photosensitive element, w w ,h w are the width and height of the target in the real world, w, h are the width and height of the image, Zw is the distance between the target object and the camera.

[0126] Step 2: Based on the distance between the camera device and the target object, the size of the object corresponding to the detected category in the real world, the distance between the camera device and the reference surface, and the boundary coordinates of the first position area, based on the assumed constraints of the small road target on the road plane, determine the second position area in the image.

[0127] Specifically, the boundary coordinates of the first location area may be the lower boundary coordinates of the first area. They may include multiple coordinate points or a single coordinate point. When the lower boundary coordinates of the first location area include a single coordinate point, the coordinate point may be the midpoint of the lower boundary or the vertex where the lower boundary intersects with another boundary (e.g., the left boundary or the right boundary).

[0128] Here, based on the determined distance between the camera and the target object, the size of the object corresponding to the detected category in the real world, the distance between the camera and the reference surface, and the boundary coordinates of the first position area, and based on the assumption that the small road target is on the ground, formula (3) can be used to inversely infer the position of the object of the category determined in step S603 in the image, which is also the second position area. Among them, formula (3) can determine the height of the object of the category presented in the image, which is the height along the gravity direction; then, based on the width-height ratio of the object of the category in the real world, the width of the object of the category presented in the image can be determined. The second position area is to use the lower boundary of the first position area as the bottom edge (for example, the midpoint of the lower boundary of the first position area is used as the midpoint of the bottom edge, or one of the vertices of the lower boundary of the first position area is used as the first starting point of the bottom edge), the width determined by formula (3) is used as the width of the left and right boundaries of the second position area, and the determined height is used as the width of the lower and upper boundaries of the second position area, so that the specific coordinate range of the second position area in the image can be determined.

[0129]

[0130] Here, (u, v) are the coordinates of a fixed point (such as the midpoint of the lower boundary or the vertex of the lower boundary) in the first position area of the image in the image coordinate system, (X w Y w Z w ) is the coordinate of a certain point of the target object in reality (such as the contact point between the pedestrian’s foot and the ground, the contact point between the car’s wheel and the ground) in the world coordinate system; f x ,f yare the focal lengths of the camera device in the x-axis and y-axis directions in the camera device coordinate system, d x ,d y are the dimensions of the photosensitive unit in the x-axis and y-axis directions on the photosensitive element in the camera coordinate system. u0, v0 are the center of the image plane (the coordinates of the pixel point at the center of the image), h w is the height of the target in the real world, h com is the distance between the camera plane and the reference plane, is the height of the image derived from the reference distance and the target height. It should be noted that when the second position area is determined using formula (3), usually a fixed point (u, v) in the first position area in the image is related to a point (X w Y w Z w ) have a mapping relationship. That is to say, the point (u, v) in the image and the point (X w Y w Z w ) are used, for example, to indicate the feet of an object, or the position of the same tire for a vehicle.

[0131] S607: Detect an error between the first location area and the second location area.

[0132] The detection of the error between the first position area and the second position area here can detect the error between the height of the first position area and the height of the second position area, or the error between the width of the first position area and the width of the second position area, or detect the error between the ratio of the width to the height of the first position area and the ratio of the width to the height of the second position area.

[0133] By determining the error between the first and second location regions, it is possible to infer the accuracy of the category of the target object detected in step S603. When the error between the first and second location regions is greater than a preset threshold, the category of the target object detected in step S603 can be considered unreliable. For example, if step S603 determines that the target object in the first location region is a person, the upper and lower boundaries of the first location region (i.e., the person's height) are 200px, and the upper and lower boundaries of the second location region (i.e., the person's height) determined based on step S605 are 400px. Alternatively, the left and right boundaries of the first location region (i.e., the person's width) are 80px, and the left and right boundaries of the second location region (i.e., the person's width) determined based on step S605 are 200px. In this case, the error between the first and second location regions is large, and the category of the target object detected in step S603 can be considered unreliable. When the error between the first and second location regions is less than the preset threshold, that is, the error between the first and second location regions is small, the category of the target object detected in step S603 can be considered reliable.

[0134] S608: Based on the error between the first location area and the second location area, correct the first confidence level to obtain a second confidence level of the category to which the target belongs.

[0135] The second confidence level here is used to indicate whether the category described by the target object is credible. When the above error is large, the second confidence level can be set to a lower value; when the above error is small, the second confidence level can be set to a higher value.

[0136] When the second confidence level is higher than a preset threshold (eg, 0.7), the category to which the target object detected in step S603 belongs can be considered credible; when the second confidence level is less than or equal to the preset threshold (eg, 0.7), the category to which the target object detected in step S603 belongs can be considered uncredible.

[0137] Combine Figure 7 、 Figure 8a 、 Figure 8b The application scenario shown is Figure 6 The implementation of steps S606 to S608 in the target detection method is described in detail.

[0138] Figure 7 is the image C captured by the camera. Assume that through the steps S601 to S605, it has been detected that a pedestrian is present in the candidate image area d in the image C with a probability of 0.6. Figure 7As can be seen from the image, the object actually represented in candidate region d is a tree. Since candidate region d is far from the camera and is a small object on the road, it is prone to misjudgment. Candidate region d here is also the first location region mentioned above. Next, the object detection device can determine the length of the object represented in candidate region d along direction U. Since step S603 detected the object represented in candidate region d as a pedestrian, the object detection device can now determine the distance between the pedestrian and the camera, assuming the object represented in candidate region d is a pedestrian. Then, the second location region where the pedestrian appears in the image can be inferred.

[0139] The second location area is determined by referring to Figure 8a-Figure 8b Assume that the focal length of the camera device is fx, fy, the world coordinate system, camera coordinate system and image coordinate system are as follows Figure 8a As shown, in the world coordinate system, the Y axis is along the direction of gravity, the Z axis is along the direction of vehicle travel, and the X axis is along the direction perpendicular to the Y axis and the Z axis. The Y axis in the world coordinate system is mapped to the V axis in the image coordinate system, and the X axis in the world coordinate system is mapped to the U axis in the image coordinate system. The X axis in the world coordinate system is mapped to the x axis in the camera coordinate system, the Y axis in the world coordinate system is mapped to the y axis in the camera coordinate system, and the Z axis in the world coordinate system is mapped to the z axis in the camera coordinate system. During the calculation process, since the image is a two-dimensional coordinate, the Z axis in the world coordinate system is not considered in the mapping process. The distance between the camera and the ground is h com .

[0140] Assuming that the target object in the candidate area c is a pedestrian, the distance between the target object and the camera device can be inferred by the above formula (1) or formula (2). Figure 8a It should be noted that the target object at position F is hypothetical and does not necessarily exist in reality. Its function is to verify the correctness of the object inference presented in step S603 for the candidate area d. Then, by querying the pre-set table, the height of the pedestrian at position F can be determined to be h w Then, by selecting Figure 8b In the real world shown in , the coordinates of the point (Xw, Yw, 0) where the pedestrian touches the ground are mapped to the point (u, v) in image C. Then, the height of the pedestrian in image C is determined to be h' by the above formula (3). The second position area in the image is as follows: Figure 8b shown.

[0141] Then, the target detection device can compare Figure 8a The first position area shown and Figure 8b The coordinate difference between the second location area is shown. Figure 8a-Figure 8b As can be seen from the figure, the difference between the first location region and the second location region is large. Therefore, the object detection device can determine a second confidence level based on the difference between the first location region and the second location region that the target object detected by the object detection model in the first location region is a pedestrian. For example, this second confidence level can be 0.1.

[0142] from Figure 6 It can be seen from the target detection method described above that Figure 3 The difference between the target detection method shown is that, in this embodiment, when the first confidence level is greater than a preset threshold, the second position area is determined, and then the misjudged target object (for example, a tree is misjudged as a pedestrian) is detected based on the error between the first position area and the second position area, thereby further improving the accuracy of small target detection on the road.

[0143] Based on the above embodiments, in some possible implementations, a step of optimizing the target detection model may also be included. Specifically, a training sample set is randomly selected, and the training sample set includes multiple training sample images. The training sample image is input into the target detection model to obtain the category and the first position area to which the object in the training sample image belongs. Then, the second position area determination method shown in step S604 is used to determine the second position area in each sample image. Finally, a second preset loss function and a back propagation algorithm are used to iteratively adjust the weights of each layer of the target detection model to optimize the target detection model. The second preset loss function is used to indicate the difference between the first position area and the second position area.

[0144] By optimizing the target detection model, the detection accuracy of the target detection model can be further improved, that is, the accuracy of road target detection can be improved, providing guarantees for subsequent autonomous driving vehicles to detect and avoid obstacles.

[0145] Please continue to refer to Figure 9 , which shows a target detection device 900 provided in an embodiment of the present application.

[0146] like Figure 9As shown, the target detection device 900 includes: an acquisition module 901, which is used to acquire an image using a camera device; a calibration module 902, which is used to calibrate the area of interest in the image based on the parameters of the camera device and a preset driving path; a first detection module 903, which is used to detect the image using a target detection algorithm to obtain the category to which the target object in the image belongs, the first position area of the target object in the image and the confidence of the category to which the target object belongs; a first correction module 904, which is used to correct the confidence of the category to which the target object belongs based on the relative position relationship between the first position area and the area of interest, and obtain a first confidence.

[0147] In addition, the target detection device 900 also includes: a determination module 905, which is used to determine the second position area in the image based on the parameters of the camera device, the boundary coordinates of the first position area in the image, and the size of the object corresponding to the preset category in the real world in response to the first confidence level being greater than a preset threshold; a second detection module 906, which is used to detect the error between the first position area and the second position area; and a second correction module 907, which is used to correct the first confidence level based on the error to obtain a second confidence level of the category to which the target belongs.

[0148] Furthermore, the parameters of the camera device include at least one of the following: the focal length of the camera device, the distance between the camera device and the reference surface, the conversion matrix from the camera device coordinate system to the image coordinate system, and the size of the photosensitive unit in the photosensitive element.

[0149] Furthermore, the determination module includes: a first determination submodule, used to determine the distance between the camera device and the target object based on the focal length of the camera device, the distance between the camera device and the reference surface, the conversion matrix from the camera device coordinate system to the image coordinate system, the size of the photosensitive unit in the photosensitive element and the boundary coordinates of the first position area in the image; a second determination submodule, used to determine the second position area in the image based on the distance between the camera device and the target object, the size of the object corresponding to the detected category in the real world, the distance between the camera device and the reference surface, and the boundary coordinates of the first position area.

[0150] Furthermore, the category to which the target object belongs is selected from the preset candidate categories based on matching results of matching the features of the target object with features of objects corresponding to a plurality of preset candidate categories.

[0151] Furthermore, the first detection module includes: a setting submodule for setting calibration parameters in a pre-trained target detection model, the calibration parameters being used to instruct the target detection model to calibrate multiple candidate areas in the image; a detection submodule for inputting the image into the target detection model to obtain an output result of the target detection model, the output result being used to indicate whether an object of a preset candidate category and the confidence of the category to which the target object belongs are presented in each of the candidate areas, wherein the target detection model is obtained by training a neural network based on training samples and calibration parameters for candidate area calibration.

[0152] Furthermore, the multiple candidate areas in the image are predetermined based on constraints; the constraints include: the area range in which the objects corresponding to each of the preset candidate categories appear in the image, and the imaging size range in which the objects corresponding to each of the preset candidate categories appear in the image.

[0153] Furthermore, the setting submodule is specifically used to: mark an initial candidate area in the image; filter the initial candidate area using the constraint conditions, and obtain the multiple candidate areas based on the filtering results.

[0154] Furthermore, the target detection device also includes a model optimization module, which is specifically used to: obtain a training sample set, the training sample set includes multiple sample images, and each of the sample images presents a target object; input the sample image into the target detection model to obtain the category to which the target object in each sample image belongs and the first position area of the target object in the sample image, and determine the second position area in each sample image based on the category to which the target object in the sample image belongs, the boundary coordinates of the first position area, and the parameters of the shooting equipment used to shoot the sample image; use a preset loss function to determine the deviation between the first position area and the second position area in each training sample, and based on the deviation, iteratively adjust the target detection model to obtain an optimized target detection model.

[0155] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0156] When integrated modules are used, the target detection device 900 may include a processor, a memory, and a communication module. The processor may control and manage the operations of the target detection device 900, for example, supporting the target detection device 900 in executing the steps performed by the various modules described above. The memory may support the target detection device 900 in executing and storing program code and data. The communication module may facilitate communication between the target detection device 900 and other devices.

[0157] Among them, the processor can implement or execute the various exemplary logic modules described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, such as a central processing unit (CPU), and can also include other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, a microcontroller or any conventional processor.

[0158] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0159] The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0160] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the above-mentioned related method steps to implement the temperature measurement method in the above-mentioned embodiment.

[0161] This embodiment further provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the temperature measurement method in the above-mentioned embodiment.

[0162] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a coupled processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to perform the above-mentioned temperature measurement method.

[0163] Among them, the processor, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0164] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices, which can be electrical, mechanical or other forms.

[0166] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0167] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0169] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A target detection method, characterized in that: The method comprises: acquiring images using a camera device; Based on the parameters of the camera device and the preset driving path, a region of interest in the image is demarcated; Detecting the image using a target detection algorithm to obtain a category to which a target object in the image belongs, a first location area of the target object in the image, and a confidence score of the category to which the target object belongs; Based on the relative positional relationship between the first position area and the region of interest, correcting the confidence level of the category to which the target object belongs to obtain a first confidence level; In response to the first confidence level being greater than a preset threshold, determining a distance between the camera and a target object based on a focal length of the camera, a distance between the camera and a reference surface, a conversion matrix from a coordinate system of the camera to an image coordinate system, a size of a photosensitive unit in a photosensitive element, and boundary coordinates of the first position area in the image; determining a second location region in the image based on a distance between the imaging device and the target object, a size of the object corresponding to the detected category in the real world, a distance between the imaging device and a reference surface, and boundary coordinates of the first location region; detecting an error between the first location area and the second location area; Based on the error, the first confidence level is corrected to obtain a second confidence level of the category to which the target object belongs.

2. The target detection method according to claim 1, wherein: The parameters of the camera device include at least one of the following: the focal length of the camera device, the distance between the camera device and a reference surface, a conversion matrix from the camera device coordinate system to the image coordinate system, and the size of a photosensitive unit in a photosensitive element.

3. The target detection method according to claim 1 or 2, characterized in that: The category to which the target object belongs is selected from the preset candidate categories based on matching results of matching the features of the target object with features of objects corresponding to a plurality of preset candidate categories.

4. The target detection method according to claim 3, wherein: The detecting the image using a target detection algorithm to obtain the category to which the target object in the image belongs, the first location area of the target object in the image, and the confidence level of the category to which the target object belongs includes: Setting calibration parameters in a pre-trained object detection model, wherein the calibration parameters are used to instruct the object detection model to calibrate a plurality of candidate regions in the image; The image is input into the target detection model to obtain an output result of the target detection model, wherein the output result is used to indicate whether an object of the preset candidate category is present in each candidate area and the confidence of the category to which the target object belongs, wherein the target detection model is obtained by training a neural network based on training samples and calibration parameters for candidate area calibration.

5. The target detection method according to claim 4, characterized in that: The plurality of candidate regions in the image are predetermined based on constraint conditions; The constraint conditions include: an area range in which objects corresponding to each of the preset candidate categories appear in the image, and an imaging size range in which objects corresponding to each of the preset candidate categories appear in the image.

6. The target detection method according to claim 5, characterized in that: The determining of a plurality of candidate regions in the image comprises: Demarcating an initial candidate region in the image; The initial candidate regions are screened using the constraint conditions, and the plurality of candidate regions are obtained based on the screening results.

7. The target detection method according to any one of claims 4 to 6, characterized in that: The method further includes an optimization step for the target detection model, wherein the optimization step includes: Acquire a training sample set, the training sample set including a plurality of sample images, each of the sample images presenting a target object; Inputting the sample images into the target detection model, obtaining the category of the target object in each sample image and the first location region of the target object in the sample image, and determining the second location region in each sample image based on the category of the target object in the sample image, the boundary coordinates of the first location region, and parameters of a camera used to capture the sample images; A preset loss function is used to determine the deviation between the first position area and the second position area in each training sample. Based on the deviation, the target detection model is iteratively adjusted to obtain an optimized target detection model.

8. A target detection device, characterized in that: include: An acquisition module, configured to acquire images using a camera device; a calibration module, configured to calibrate a region of interest in the image based on parameters of the camera device and a preset driving path; a first detection module, configured to detect the image using a target detection algorithm to obtain a category to which a target object in the image belongs, a first location area of the target object in the image, and a confidence score of the category to which the target object belongs; a first correction module, configured to correct the confidence level of the category to which the target object belongs based on the relative positional relationship between the first position area and the region of interest, to obtain a first confidence level; The determination module includes a first determination submodule and a second determination submodule, wherein the first determination submodule is configured to determine, in response to the first confidence level being greater than a preset threshold, the distance between the camera device and the target object based on the focal length of the camera device, the distance between the camera device and a reference surface, a conversion matrix from the camera device coordinate system to the image coordinate system, the size of a photosensitive unit in a photosensitive element, and the boundary coordinates of the first position area in the image; and the second determination submodule is configured to determine a second position area in the image based on the distance between the camera device and the target object, the size of an object corresponding to the detected category in the real world, the distance between the camera device and the reference surface, and the boundary coordinates of the first position area. A second detection module, configured to detect an error between the first location area and the second location area; A second correction module is used to correct the first confidence level based on the error to obtain a second confidence level of the category to which the target belongs.

9. The target detection device according to claim 8, characterized in that The parameters of the camera device include at least one of the following: the focal length of the camera device, the distance between the camera device and a reference surface, a conversion matrix from the camera device coordinate system to the image coordinate system, and the size of a photosensitive unit in a photosensitive element.

10. The target detection device according to claim 8 or 9, characterized in that: The category to which the target object belongs is selected from the preset candidate categories based on a matching result obtained by matching features of the target object with features of objects corresponding to a plurality of preset candidate categories.

11. The target detection device according to claim 10, wherein: The first detection module includes: A setting submodule is used to set calibration parameters in a pre-trained object detection model, wherein the calibration parameters are used to instruct the object detection model to calibrate multiple candidate regions in the image; a detection submodule, configured to input the image into the target detection model and obtain an output result of the target detection model, wherein the output result is used to indicate whether an object of a preset candidate category is present in each candidate area and the confidence level of the category to which the target object belongs, wherein the target detection model is obtained by training a neural network based on training samples and calibration parameters for calibrating candidate areas.

12. The target detection device according to claim 11, wherein: The plurality of candidate regions in the image are predetermined based on constraint conditions; The constraint conditions include: an area range in which objects corresponding to each of the preset candidate categories appear in the image, and an imaging size range in which objects corresponding to each of the preset candidate categories appear in the image.

13. The target detection device according to claim 12, wherein: The setting submodule is specifically used for: Demarcating an initial candidate region in the image; The initial candidate regions are screened using the constraint conditions, and the plurality of candidate regions are obtained based on the screening results.

14. The target detection device according to any one of claims 11 to 13, characterized in that: The target detection device further includes a model optimization module, which is specifically configured to: Acquire a training sample set, the training sample set including a plurality of sample images, each of the sample images presenting a target object; Inputting the sample images into the target detection model, obtaining the category of the target object in each sample image and the first location region of the target object in the sample image, and determining the second location region in each sample image based on the category of the target object in the sample image, the boundary coordinates of the first location region, and parameters of a camera used to capture the sample images; A preset loss function is used to determine the deviation between the first position area and the second position area in each training sample. Based on the deviation, the target detection model is iteratively adjusted to obtain an optimized target detection model.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 7.

16. A readable storage medium, characterized in that The method comprises computer instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 7.

17. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

Images