Object detection method

Abstract

Embodiments of the present specification provide an object detection method, comprising: determining a to-be-detected image containing a target object; inputting the to-be-detected image into an object detection model to obtain object probability information of each pixel point in the to-be-detected image relative to the target object and position information of the each pixel point relative to a target object detection region corresponding to the each pixel point; and determining a target object detection region of the target object in the to-be-detected image according to the object probability information and the position information. In combination of the object probability information and the position information, when determining the target object detection region, not only whether a pixel point is a target object is considered, but also position information of each pixel point relative to a target object detection region corresponding to the each pixel point is obtained, so that the determined target object detection region can be more accurate, thereby improving the accuracy of the detection result.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to object detection methods. Background Technology

[0002] With the development of computer technology, pedestrian detection technology has emerged. Pedestrian detection can be understood as using computer vision technology to determine whether pedestrians exist in an image or video sequence and to locate them accurately. As a type of object detection, pedestrian detection can be used to identify and locate pedestrians in images or videos. It can serve as a preliminary step for pedestrian tracking, pedestrian recognition, and human attribute analysis, and is widely used in video surveillance, intelligent transportation, and other fields.

[0003] Typically, during pedestrian detection, the presence of significant background elements beyond pedestrians in images or video sequences can negatively impact detection results, leading to inaccurate findings. Therefore, an effective technical solution is urgently needed to address this issue. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide an object detection method. One or more embodiments of this specification also relate to an object detection device, a method for training an object detection model, a training device for an object detection model, two pedestrian detection methods, two pedestrian detection devices, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an object detection method is provided, comprising:

[0006] Identify the image to be detected that contains the target object;

[0007] The image to be detected is input into the object detection model to obtain the object probability information of each pixel relative to the target object and the position information of each pixel relative to its corresponding target object detection region in the multiple pixels of the image to be detected.

[0008] Based on the object probability information and the location information, the target object detection region of the target object is determined in the image to be detected.

[0009] According to a second aspect of the embodiments of this specification, an object detection apparatus is provided, comprising:

[0010] The first determining module is configured to determine the image to be detected that contains the target object;

[0011] The input module is configured to input the image to be detected into the object detection model to obtain object probability information of each pixel relative to the target object and position information of each pixel relative to its corresponding target object detection region in multiple pixels of the image to be detected.

[0012] The second determining module is configured to determine the target object detection region of the target object in the image to be detected based on the object probability information and the location information.

[0013] According to a third aspect of the embodiments of this specification, a method for training an object detection model is provided, comprising:

[0014] Identify image samples containing the target object, and determine the target object detection region label in the image samples;

[0015] Based on the target object detection region label, determine the object probability information label of each pixel relative to the target object and the position information label of each pixel relative to its corresponding target object detection region label in the multiple pixels of the image sample;

[0016] The image sample is input into the object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, which are output by the object detection model.

[0017] The object detection model is trained based on the predicted object probability information and the object probability information label, the predicted location information and the location information label, until an object detection model that meets the training stopping condition is obtained.

[0018] According to a fourth aspect of the embodiments of this specification, a training apparatus for an object detection model is provided, comprising:

[0019] The first determining module is configured to determine an image sample containing a target object, and to determine a target object detection region label of the target object in the image sample;

[0020] The second determining module is configured to determine, based on the target object detection region label, the object probability information label of each pixel relative to the target object and the position information label of each pixel relative to its corresponding target object detection region label in the multiple pixels of the image sample;

[0021] The input module is configured to input the image sample into the object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, which are output by the object detection model.

[0022] The training module is configured to train the object detection model based on the predicted object probability information and the object probability information label, the predicted location information and the location information label, until an object detection model that meets the training stopping condition is obtained.

[0023] According to a fifth aspect of the embodiments of this specification, a pedestrian detection method is provided, applied to a cloud-based device, comprising:

[0024] A pedestrian detection request sent by a receiving end-side device, wherein the pedestrian detection request carries an image to be detected;

[0025] The image to be detected is input into the object detection model to obtain the object probability information of each pixel relative to the pedestrian and the position information of each pixel relative to its corresponding pedestrian detection area in the multiple pixels of the image to be detected.

[0026] Based on the object probability information and the location information, determine the pedestrian detection region of the pedestrian in the image to be detected;

[0027] Based on the pedestrian detection area, the image to be detected is processed to obtain the processed target image;

[0028] The target image is sent to the end-side device.

[0029] According to a sixth aspect of the embodiments of this specification, a pedestrian detection device is provided, applied to cloud-based equipment, comprising:

[0030] The receiving module is configured to receive pedestrian detection requests sent by the end-side device, wherein the pedestrian detection request carries an image to be detected;

[0031] The input module is configured to input the image to be detected into the object detection model to obtain object probability information of each pixel relative to the pedestrian and position information of each pixel relative to its corresponding pedestrian detection area in multiple pixels of the image to be detected.

[0032] The determination module is configured to determine the pedestrian detection region of the pedestrian in the image to be detected based on the object probability information and the location information;

[0033] The processing module is configured to process the image to be detected based on the pedestrian detection area to obtain the processed target image;

[0034] The sending module is configured to send the target image to the end-side device.

[0035] According to a seventh aspect of the embodiments of this specification, a pedestrian detection method is provided, applied to a cloud-based device, comprising:

[0036] The vehicle's camera receives road images captured by the vehicle's camera during its movement.

[0037] The road image is input into the object detection model to obtain the object probability information of each pixel relative to the pedestrian in multiple pixels of the road image, and the position information of each pixel relative to its corresponding pedestrian detection area.

[0038] Based on the object probability information and the location information, determine the pedestrian detection area of ​​the pedestrian in the road image;

[0039] The vehicle's trajectory is determined based on the pedestrian detection area;

[0040] The vehicle is controlled according to the driving trajectory.

[0041] According to an eighth aspect of the embodiments of this specification, a pedestrian detection device is provided, applied to cloud-based equipment, comprising:

[0042] The receiving module is configured to receive road images captured by the vehicle's camera device during the vehicle's movement.

[0043] The input module is configured to input the road image into the object detection model to obtain object probability information of each pixel relative to a pedestrian and position information of each pixel relative to its corresponding pedestrian detection area in multiple pixels of the road image.

[0044] The first determining module is configured to determine the pedestrian detection region of the pedestrian in the road image based on the object probability information and the location information;

[0045] The second determining module is configured to determine the vehicle's trajectory based on the pedestrian detection area;

[0046] The control module is configured to control the vehicle based on the driving trajectory.

[0047] According to a ninth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0048] Memory and processor;

[0049] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0050] According to a tenth aspect of an embodiment of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the method described above.

[0051] According to an eleventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method.

[0052] One embodiment of this specification provides an object detection method, which determines an image to be detected containing a target object; inputs the image to be detected into an object detection model to obtain object probability information of each pixel relative to the target object and position information of each pixel relative to its corresponding target object detection region among multiple pixels of the image to be detected; and determines the target object detection region of the target object in the image to be detected based on the object probability information and the position information.

[0053] The above method processes the image to be detected using an object detection model, which can obtain the object probability information of each pixel in the image relative to the target object. This allows the method to determine the probability that each pixel in the image is a target object. Furthermore, it can obtain the positional information of each pixel in the image relative to its corresponding target object detection region. By combining the object probability information and the positional information, the method not only considers the probability of a pixel being a target object when determining the target object detection region, but also obtains the positional information of each pixel relative to its corresponding target object detection region. This makes the determined target object detection region more accurate, thereby improving the accuracy of the detection results. Attached Figure Description

[0054] Figure 1 This is a schematic diagram illustrating an application scenario of an object detection method provided in one embodiment of this specification;

[0055] Figure 2 This is a flowchart of an object detection method provided in one embodiment of this specification;

[0056] Figure 3 This is a flowchart illustrating the training process of an object detection model in an object detection method provided in one embodiment of this specification.

[0057] Figure 4 This is a schematic diagram of an object detection model in an object detection method provided in one embodiment of this specification;

[0058] Figure 5 This is a schematic diagram of the classification adjustment network layer of the object detection model in an embodiment of the object detection method provided in this specification;

[0059] Figure 6 This is a schematic diagram of the feature compensation network layer of an object detection model in an embodiment of the object detection method provided in this specification;

[0060] Figure 7 This is a flowchart illustrating the processing procedure of an object detection method provided in one embodiment of this specification.

[0061] Figure 8 This is a schematic diagram of the structure of an object detection device provided in one embodiment of this specification;

[0062] Figure 9 This is a flowchart illustrating a training method for an object detection model provided in one embodiment of this specification;

[0063] Figure 10 This is a schematic diagram of the structure of a training device for an object detection model provided in one embodiment of this specification;

[0064] Figure 11 This is a flowchart of a pedestrian detection method provided in one embodiment of this specification;

[0065] Figure 12 This is a schematic diagram of the structure of a pedestrian detection device provided in one embodiment of this specification;

[0066] Figure 13 This is a flowchart of another pedestrian detection method provided in one embodiment of this specification;

[0067] Figure 14 This is a schematic diagram of another pedestrian detection device provided in one embodiment of this specification;

[0068] Figure 15 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0069] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0070] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0071] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0072] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0073] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0074] CPN: Complete Proposal Network, used for feature compensation.

[0075] C2F: Coarse to Fine.

[0076] resnet50: is a network layer used for feature extraction.

[0077] CNN: Convolutional Neural Network.

[0078] Focal loss: A loss function that reduces the weight of easily classified samples, allowing the model to focus more on difficult-to-classify samples during training.

[0079] This specification provides an object detection method, and also relates to an object detection device, an object detection model training method, an object detection model training device, two pedestrian detection methods, two pedestrian detection devices, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0080] See Figure 1 , Figure 1 A schematic diagram illustrating an application scenario of an object detection method provided according to an embodiment of this specification is shown.

[0081] Figure 1 The system includes a vehicle 102 and a cloud-based device 104. The dashcam installed on the vehicle 102 serves as a recording device and can communicate with the cloud-based device 104.

[0082] During the driving of vehicle 102, in order to achieve unmanned driving of vehicle 102, cloud-based device 104 can control the driving of vehicle 102 based on the footage captured by the dashcam.

[0083] In practice, the dashcam of vehicle 102 can send the captured image of the front of vehicle 102 to cloud-based device 104. After receiving the image, cloud-based device 104 can first determine whether the image includes a pedestrian. If it determines that the image includes a pedestrian, it can input the image into an object detection model to obtain the probability of each pixel in the image being a pedestrian, as well as the distance of each pixel relative to the pedestrian detection area, output by the object detection model. Based on this probability and distance, the position of the pedestrian in the image is determined, i.e., the pedestrian detection area is determined in the image. Based on this detection area, the driving trajectory of vehicle 102 is planned, thereby controlling the driving of vehicle 102 according to the driving trajectory.

[0084] The above method uses an object detection model to detect and locate pedestrians, thereby enabling control of autonomous vehicles and ensuring the safety of autonomous driving.

[0085] See Figure 2 , Figure 2 A flowchart of an object detection method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0086] Step 202: Determine the image to be detected that contains the target object.

[0087] Specifically, the object detection method provided in the embodiments of this specification can be used in the field of pedestrian detection. It can detect pedestrians in image or video sequences, facilitating subsequent intelligent monitoring of target areas and trajectory prediction for autonomous vehicles based on the pedestrian detection results. Furthermore, this object detection method can also be used in other fields, such as detecting entities like buildings, grass, and animals from images or videos as needed. The embodiments of this specification do not limit this application.

[0088] For ease of understanding, the embodiments in this specification all use the application of the object detection method to a pedestrian detection scenario as an example for detailed description, but this does not affect the implementation of the object detection method in other feasible scenarios.

[0089] In this context, the target object can be understood as the object to be detected within the image. For example, in the field of pedestrian detection, the target object can be understood as the pedestrian to be detected.

[0090] In practical applications, the image to be detected can be obtained by a camera or a video frame obtained from a video recording. For example, in the field of autonomous driving, the vehicle's camera (such as a dashcam) can capture images or videos of the area in front of the vehicle.

[0091] In one embodiment of this specification, in order to determine whether an image to be detected contains a target object, multiple images to be detected captured by the imaging device can be scanned. Based on the scanning results, an image to be detected containing the target object is determined among the multiple images to be detected, which facilitates the subsequent determination of the position of the target object in the image to be detected according to the object detection method.

[0092] In another embodiment of this specification, multiple images to be detected captured by the imaging device can be directly input into the object detection model, and the object detection model can be used to determine whether the image to be detected includes a target object.

[0093] Step 204: Input the image to be detected into the object detection model to obtain the object probability information of each pixel relative to the target object and the position information of each pixel relative to its corresponding target object detection region in the multiple pixels of the image to be detected.

[0094] Specifically, after determining the image to be detected, the image can be input into the object detection model to obtain the object probability information of each pixel relative to the target object and the position information of each pixel relative to its corresponding target object detection region.

[0095] In this context, a pixel can be understood as the smallest unit of an image, and an image is composed of multiple pixels. Therefore, the multiple pixels in the image to be detected can be understood as the multiple pixels that constitute the image. The object probability information of each pixel relative to the target object can be understood as the probability value of whether each pixel in the image to be detected is the target object. For example, a probability of 1 indicates that the pixel is the target object, and a probability of 0 indicates that the pixel is not the target object. In the field of pedestrian detection, an image can be divided into multiple pixels. When the object probability information of a pixel is 1, it means that the pixel corresponds to a pedestrian in the image; when the object probability information of a pixel is 0, it means that the pixel corresponds to a pedestrian in the image. The target object detection region corresponding to a pixel can be understood as the detection bounding box of the target object corresponding to the pixel. The positional information of each pixel relative to its corresponding target object detection region can be understood as the distance between each pixel and its corresponding target object detection region, such as the distance between the top, bottom, left, and right borders of the target object detection region. This positional information can be used to determine the target object detection region of the target object in the image to be detected.

[0096] Based on this, the image to be detected can be input into the object detection model, and the object detection model can be used to process the image to obtain the probability of each pixel being a target object among multiple pixels in the image to be detected, and the distance between each pixel and its corresponding target object detection box.

[0097] Understandably, regarding the positional information of each pixel relative to its corresponding target object detection region, let's take an image containing two target objects as an example: target object 1 is located in target object detection region 1, and target object 2 is located in target object detection region 2. The target object detection region corresponding to the pixel of target object 1 is region 1, and the target object detection region corresponding to the pixel of target object 2 is region 2. For pixels that are neither target object 1 nor target object 2, these pixels can be background pixels in the image to be detected, and the positional information of the target object detection region corresponding to these pixels can be represented as null values.

[0098] In practical applications, the object probability information of each pixel relative to the target object can be represented in the form of a pixel feature map. Similarly, the positional information of each pixel relative to its corresponding target object detection region can also be represented in the form of a pixel feature map. In essence, object probability information and positional information can be understood as pixel features of the image to be detected.

[0099] For example, in a pedestrian detection scenario, if it is determined that the image to be detected contains pedestrians, the image to be detected can be input into an object detection model to obtain the probability of each pixel being a pedestrian and the distance between each pixel and its corresponding pedestrian detection box.

[0100] In practical applications, in order to ensure that the object detection model can detect the specific location of the target object in the image to be detected, the object detection model needs to be trained before the image to be detected is input into the object detection model so that it meets the usage requirements. Figure 3 This diagram illustrates a flowchart of the training process of an object detection model in an object detection method according to an embodiment of this specification. (See also...) Figure 3 The specific training process for the object detection model is described in steps 302 to 308 below.

[0101] Step 302: Determine an image sample containing the target object, and determine the target object detection region label in the image sample.

[0102] Specifically, in the training dataset, image samples containing the target object can be identified, and the target object detection region label can be determined within these image samples. This target object detection region label can be understood as the detection box of the target object in the image sample, and this detection box is used as the training label.

[0103] Step 304: Based on the target object detection region label, determine the object probability information label of each pixel relative to the target object and the position information label of each pixel relative to its corresponding target object detection region label in the multiple pixels of the image sample.

[0104] Specifically, based on the target object detection region label, the object probability information of pixels within the target object detection region label in the image sample can be set to 1, and the object probability information of pixels outside the target object detection region label can be set to 0. This serves as the object probability information label for each pixel relative to the target object. Correspondingly, the distance of each pixel relative to its corresponding target object detection region label can be determined based on the target object detection region label.

[0105] Step 306: Input the image sample into the object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, which are output by the object detection model.

[0106] In specific implementation, the step of inputting the image sample into the object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, output by the object detection model, includes:

[0107] The image sample is input into an object detection model. In the object detection model, a feature extraction layer is used to extract features from the image sample to obtain an initial pixel feature map, wherein the initial pixel feature map includes multiple pixels of the image sample.

[0108] The initial pixel feature map is processed using a regression network layer to obtain the initial predicted position information of each pixel relative to its corresponding target object detection region among multiple pixels of the image sample.

[0109] The initial predicted position information is processed using a convolutional layer to obtain the predicted position information of each pixel relative to its corresponding target object detection region among multiple pixels of the image sample.

[0110] The initial pixel feature map is processed using a classification network layer to obtain the initial predicted object probability information of each pixel relative to the target object in multiple pixels of the image sample.

[0111] Based on the initial predicted location information and the initial predicted object probability information, the predicted object probability information of each pixel in the image sample relative to the target object is determined.

[0112] The object detection model includes a feature extraction layer, a regression network layer, a convolutional layer, and a classification network layer. The feature extraction layer extracts pixel features from the image. The regression network layer determines the positional information of each pixel relative to its corresponding target object detection region. The classification network layer determines the object probability information of each pixel relative to the target object. The convolutional layer performs convolutional processing on the pixel features of the image.

[0113] In practical applications, the feature extraction layer can be, for example, a ResNet50 network structure layer. Alternatively, the feature extraction layer can be any other neural network layer used for feature extraction. This specification does not limit the specific implementation of this method.

[0114] Since the predicted location information is the output of the object detection model, the dimensions of the initial predicted location information and the predicted location information are different for easy visualization. For example, the dimension of the initial predicted location information can be 256, while the dimension of the predicted location information can be 4.

[0115] For details, see Figure 4 , Figure 4 A schematic diagram of an object detection model in an object detection method according to an embodiment of this specification is shown. Figure 4 As shown, after inputting the image sample into the object detection model 400, the image sample can be input into the feature extraction layer 402. The feature extraction layer 402 performs feature extraction processing on the image sample to obtain an initial pixel feature map. This initial pixel feature map is then input into the regression network layer 404, which processes it to obtain initial predicted position information for each pixel relative to its corresponding target object detection region. This initial predicted position information is then input into the convolutional layer 406 to obtain the output of the convolutional layer 406, which shows the predicted position information for each pixel relative to its corresponding target object detection region. This predicted position information can be used as the output of the object detection model 400. Alternatively, the initial pixel feature map can be input into the classification network layer 408 to obtain the output of the classification network layer 408, which shows the initial predicted object probability information for each pixel relative to the target object. Based on the initial predicted location information and the initial predicted object probability information, the initial predicted object probability information is adjusted and optimized using the initial predicted location information. Finally, the predicted object probability information of each pixel in the image sample relative to the target object is obtained. This predicted object probability information can be used as the output of the object detection model 400.

[0116] In summary, by determining the initial predicted location information and the initial object probability information separately, it is easier to optimize and adjust the initial object probability information based on the initial predicted location information, so that the final predicted object probability information is more accurate.

[0117] In specific implementation, determining the predicted object probability information of each pixel in the image sample relative to the target object based on the initial predicted location information and the initial predicted object probability information includes:

[0118] The probability information of the initial predicted object is processed by the classification adjustment network layer to obtain the probability information of the first intermediate predicted object;

[0119] The initial predicted object probability information and the initial predicted location information are processed using a feature compensation network layer to obtain the second intermediate predicted object probability information;

[0120] Based on the first intermediate predicted object probability information and the second intermediate predicted object probability information, the predicted object probability information of each pixel in the image sample relative to the target object is determined.

[0121] The object detection model also includes a classification adjustment network layer and a feature compensation network layer.

[0122] In practical applications, the classification adjustment network layer can be, for example, a C2F network layer. This C2F network layer can be used to adjust the initial predicted object probability information. For instance, an image sample may contain 5 pixels, and the initial predicted object probability information for each pixel relative to the target object is 1, 1, 0, 1, 0, respectively. An initial predicted object probability of 1 indicates a positive sample, meaning the pixel is the target object. An initial predicted object probability of 0 indicates a negative sample, meaning the pixel is not the target object. This C2F network layer can adjust the initial predicted object probability information for each pixel relative to the target object so that the final output first intermediate predicted object probability information contains only one positive sample, such as 1, 0, 0, 0, 0. The feature compensation network layer can be, for example, a CPN network, used to perform additional feature compensation on the initial predicted object probability information based on the initial predicted location information.

[0123] Based on this, such as Figure 4 As shown, when determining the predicted object probability information of each pixel in an image sample relative to the target object based on the initial predicted location information and the initial predicted object probability information, the initial predicted object probability information can be input into the classification adjustment network layer 410 to obtain the first intermediate predicted object probability information output by the classification adjustment network layer 410. This initial predicted object probability information and the initial predicted location information are then input into the feature compensation network layer 412 to obtain the second intermediate predicted object probability information output by the feature compensation network layer 412. Based on this first and second intermediate predicted object probability information, the predicted object probability information of each pixel in the image sample relative to the target object can be determined.

[0124] Furthermore, when determining the final prediction object probability information based on the first intermediate prediction object probability information and the second intermediate prediction object probability information, the Hadman product can be used to calculate the final prediction object probability information, thereby ensuring the combination of the first intermediate prediction object probability information and the second intermediate prediction object probability information.

[0125] In summary, by setting up a classification adjustment network layer and a feature compensation network layer, additional compensation is introduced while ensuring the uniqueness of the positive samples output, enabling the object detection model to recall more difficult samples that were not identified in the classification adjustment network layer.

[0126] Specifically, in order to produce a final positive output only for a ground truth instance after the classification network layer outputs the initial predicted object probability information, the number of positive samples allocated to that ground truth instance can be gradually reduced in the classification adjustment network layer. The specific implementation method is as follows:

[0127] The classification adjustment network layer includes n classification adjustment units;

[0128] Accordingly, the step of inputting the initial predicted object probability information into the classification adjustment network layer and processing the initial predicted object probability information using the classification adjustment network layer to obtain the first intermediate predicted object probability information includes:

[0129] The initial predicted object probability information is input into the i-th classification adjustment unit to obtain the i-th predicted object probability information output by the i-th classification adjustment unit, where the initial value of i is 1, i∈[1,n], and n is a positive integer;

[0130] The probability information of the i-th predicted object is input into the (i+1)-th classification adjustment unit to obtain the probability information of the (i+1)-th predicted object output by the (i+1)-th classification adjustment unit.

[0131] Determine if i+1 is greater than or equal to n;

[0132] If not, increment i by 1 and continue to execute the step of inputting the probability information of the i-th predicted object into the (i+1)-th classification adjustment unit;

[0133] If so, the probability information of the (i+1)th predicted object is determined as the probability information of the first intermediate predicted object.

[0134] The classification adjustment unit can be understood as a CNN convolutional layer.

[0135] Specifically, the classification adjustment network layer may include multiple classification adjustment units. The initial predicted object probability information is input into the first classification adjustment unit in the classification adjustment network layer to obtain the first predicted object probability information output by the first classification adjustment unit. The first predicted object probability information is input into the second classification adjustment unit to obtain the second predicted object probability information output by the second classification adjustment unit, and so on. The probability information of the (i+1)th predicted object output by the last classification adjustment unit is determined as the first intermediate predicted object probability information.

[0136] Furthermore, a classification loss function is applied to each classification adjustment unit, and the classification adjustment unit is trained based on this classification loss function. The specific implementation method is as follows:

[0137] According to a preset label allocation strategy, intermediate object probability information labels are determined for each classification adjustment unit, wherein the preset label allocation strategy is to decrease the number of positive samples included in the intermediate object probability information labels allocated to each classification adjustment unit in turn.

[0138] Each classification adjustment unit is trained based on the predicted object probability information output by each classification adjustment unit and the intermediate object probability information label of each classification adjustment unit.

[0139] For details, see Figure 5 , Figure 5 A schematic diagram of a classification adjustment network layer in an object detection model according to an embodiment of this specification is shown. Taking an example where the classification adjustment network layer includes three classification adjustment units, the initial predicted object probability information is input to the first classification adjustment unit 502 to obtain the first predicted object probability information output by the first classification adjustment unit 502. The first predicted object probability information is input to the second classification adjustment unit 504 to obtain the second predicted object probability information output by the second classification adjustment unit 504. The second predicted object probability information is input to the third classification adjustment unit 506 to obtain the third predicted object probability information output by the third classification adjustment unit 506. This third predicted object probability information is used as the first intermediate predicted object probability information finally output by the classification adjustment network layer.

[0140] For these three classification adjustment units, the preset label allocation strategy is to progressively decrease the number of positive samples included in the intermediate object probability information labels assigned to each unit. For example, the intermediate object probability information labels assigned to the first classification adjustment unit 502 include 10 positive samples, those assigned to the second unit 504 include 5 positive samples, and those assigned to the third unit 506 include 1 positive sample. For each classification adjustment unit, a loss value can be calculated based on the predicted object probability information output by each unit and the intermediate object probability information labels assigned to each unit. The unit is then trained based on this loss value.

[0141] In practical applications, the loss function used to calculate the loss value can be the focal loss function. The overall optimization objective function L for this classification-adjusting network layer is... c2f As shown in the following formula (1).

[0142]

[0143] Where i is the sequence number of the classification adjustment unit, such as the first classification adjustment unit, the second classification adjustment unit, etc. n is the number of classification adjustment units included in the classification adjustment network layer. Li is the focal loss function applied to the i-th classification adjustment unit.

[0144] Furthermore, in practical applications, a one-to-Mi label allocation strategy can be adopted for the i-th classification adjustment unit. As i increases, Mi will monotonically decrease, thus achieving label allocation from coarse-grained to fine-grained.

[0145] In summary, by assigning different labels to each classification adjustment unit, the number of training positive samples is gradually reduced, enabling the object detection model to adaptively refine the classification decision boundary for learning and output accurate detection results, thus reducing ambiguity issues during the training and testing phases.

[0146] In practical applications, to further improve the detection performance of object detection models for difficult samples, a feature compensation network layer can be introduced to provide additional information compensation for difficult samples. The specific implementation method is as follows:

[0147] The step of processing the initial predicted object probability information and the initial predicted position information using the feature compensation network layer to obtain the second intermediate predicted object probability information includes:

[0148] The feature compensation network layer is used to concatenate the initial predicted object probability information and the initial predicted location information to obtain the first predicted feature.

[0149] The feature compensation network layer is used to concatenate and enhance the initial predicted object probability information and the initial predicted location information to obtain the second predicted feature.

[0150] The feature compensation network layer is used to process the initial predicted object probability information and the initial predicted location information to obtain a third predicted feature;

[0151] Based on the first prediction feature, the second prediction feature, and the third prediction feature, the probability information of the second intermediate prediction object is determined.

[0152] In practical applications, concatenating the initial predicted object probability information and the initial predicted location information can be understood as performing feature concatenation on the initial predicted object probability information and the initial predicted location information in the channel dimension. Feature enhancement processing can be understood as multi-scale feature enhancement. Specifically, in the process of multi-scale feature enhancement, max pooling can be performed on the initial predicted object probability information and the initial predicted location information in three-dimensional space.

[0153] In summary, by processing the initial predicted object probability and the initial predicted location information differently in the feature compensation network layer, and determining the second intermediate predicted object probability information based on the multiple predicted features obtained after different processing, it is possible to enhance and compensate for these two features, thereby enriching the features and further improving the accuracy of the model's output.

[0154] Furthermore, in the feature compensation network layer, the above three processing procedures yield three predicted features (i.e., the first predicted feature, the second predicted feature, and the third predicted feature). In order to save model computing power, only one processing procedure can be performed to obtain one predicted feature (such as obtaining the first predicted feature or obtaining the second predicted feature, etc.), or two processing procedures can be performed to obtain two predicted features (such as obtaining the first predicted feature and the second predicted feature, or obtaining the first predicted feature and the third predicted feature, etc.). The embodiments in this specification do not limit this.

[0155] In specific implementation, the step of processing the initial predicted object probability information and the initial predicted location information using the feature compensation network layer to obtain the third predicted feature includes:

[0156] The feature compensation network layer is used to concatenate the initial predicted object probability information and the initial predicted location information to obtain the first concatenated feature.

[0157] The initial predicted position information is inverted, and the initial predicted position information and the initial predicted object probability information obtained after inversion are concatenated and convolved to obtain the second concatenated feature.

[0158] The first and second spliced ​​features are subjected to feature enhancement processing to obtain the third predicted feature.

[0159] The negation process can be understood as the Neg negation operation. Neg can be understood as the complement instruction in assembly instructions, which performs a complement operation on the operand.

[0160] Specifically, when processing the initial predicted object probability information and initial predicted position information using the feature compensation network layer to obtain the third predicted feature, the initial predicted object probability information and initial predicted position information can be concatenated in the feature compensation network layer to obtain the first concatenated feature. The initial predicted position information is then inverted. The initial predicted position information and initial predicted object probability information obtained after inversion are then concatenated and convolved to obtain the second concatenated feature. The first and second concatenated features are then enhanced to obtain the third predicted feature.

[0161] In summary, by inverting the initial predicted location information, the initial predicted location information is further enriched, the diversity of features is ensured, and the processing capability of the model is further improved.

[0162] In specific implementation, determining the probability information of the second intermediate prediction object based on the first prediction feature, the second prediction feature, and the third prediction feature includes:

[0163] Using a preset activation function, the first prediction feature, the second prediction feature, and the third prediction feature are normalized and convolved to obtain the probability information of the second intermediate prediction object.

[0164] In practical applications, when normalizing the first, second, and third prediction features, over-normalization can be performed on them.

[0165] Specifically, in order to achieve the fusion of the first, second, and third prediction features and thus determine the probability information of the second intermediate prediction object, a preset activation function can be used to perform group normalization and convolution processing on the first, second, and third prediction features to obtain the probability information of the second intermediate prediction object.

[0166] In addition, the fusion of the first predictive feature, the second predictive feature and the third predictive feature can also be achieved through other fusion methods, such as through splicing processing. This specification does not limit the embodiments in this way.

[0167] In summary, by fusing the first, second, and third predictive features, the obtained probability information of the second intermediate predicted object is enriched, ensuring its diversity and further improving the model's processing performance.

[0168] Specifically, Figure 6 A schematic diagram of the feature compensation network layer of an object detection model in an object detection method according to an embodiment of this specification is shown. Figure 6As shown, the feature compensation network layer includes a splicing unit 602, a feature enhancement unit 604, a convolutional unit 606, an inversion unit 608, and a normalization unit 610. Specifically, the splicing unit 602 splices the initial predicted object probability information and the initial predicted position information to obtain the first predicted feature F1. The initial predicted object probability information and the initial predicted position information pass through the splicing unit 602, the convolutional unit 606, and the feature enhancement unit 604 in sequence to obtain the second predicted feature F2. The initial predicted position information and the initial predicted object probability information pass through the splicing unit 602, or the first spliced ​​feature. The initial predicted position information passes through the inversion unit 608, and the inverted initial predicted position information and the initial predicted object probability information pass through the splicing unit 602 and the convolutional unit 606 in sequence to obtain the second spliced ​​feature. The first spliced ​​feature and the second spliced ​​feature pass through the feature enhancement unit 604 to obtain the third predicted feature F3. The first prediction feature F1, the second prediction feature F2, and the third prediction feature F3 are processed by the normalization unit 610 and the convolution unit 606 to obtain the probability information of the second intermediate prediction object.

[0169] Step 308: Train the object detection model based on the predicted object probability information and the object probability information label, the predicted location information and the location information label, until an object detection model that meets the training stopping condition is obtained.

[0170] Specifically, the first model loss value can be calculated based on the predicted object probability information and the object probability information label, and the second model loss value can be calculated based on the predicted location information and the location information label. The object detection model can be trained based on the first model loss value and the second model loss value until an object detection model that meets the training stopping condition is obtained.

[0171] Among them, the training stopping condition can be, for example, reaching a preset threshold for the number of training iterations, or it can be that the model loss value reaches a preset threshold for the loss value.

[0172] In addition, the object probability information label can include only one positive sample, with the remaining pixels treated as negative samples.

[0173] Step 206: Determine the target object detection region in the image to be detected based on the object probability information and the location information.

[0174] Specifically, after determining the object probability information and location information in step 204, the target object detection area can be determined.

[0175] In specific implementation, determining the target object detection region in the image to be detected based on the object probability information and the location information includes:

[0176] Based on the object probability information of each pixel relative to the target object, the target pixel corresponding to the target object is determined;

[0177] Based on the position information of each pixel relative to its corresponding target object detection area, the target position information of the target pixel relative to its corresponding target object detection area is determined.

[0178] Based on the target location information, the target object detection region is determined in the image to be detected.

[0179] Specifically, based on the object probability information of each pixel, if the object probability information is greater than a preset probability threshold, the pixel is determined to be the target pixel corresponding to the target object. Furthermore, the target position information of the target pixel relative to its corresponding target object detection area is determined from the position information of each pixel relative to its corresponding target object detection area, thereby determining the target object detection area of ​​the target object.

[0180] In summary, the above method processes the image to be detected using an object detection model, obtaining the object probability information of each pixel in the image relative to the target object. This allows for the determination of the probability that each pixel in the image is a target object. Furthermore, it can obtain the positional information of each pixel relative to its corresponding target object detection region. By combining the object probability information and the positional information, the determination of the target object detection region not only considers the probability of a pixel being a target object but also obtains the positional information of each pixel relative to its corresponding target object detection region. This makes the determined target object detection region more accurate, thereby improving the accuracy of the detection results.

[0181] The following is in conjunction with the appendix Figure 7 Taking the application of the object detection method provided in this specification in pedestrian detection as an example, the object detection method will be further explained. Figure 7 The present specification shows a flowchart of the processing procedure of an object detection method according to an embodiment, which specifically includes the following steps.

[0182] Step 702: Determine the image to be detected that contains the target object, and input the image to be detected into the object detection model.

[0183] Step 704: In the object detection model, the feature extraction layer is used to extract features from the image to be detected, and the initial pixel feature map of the image to be detected is obtained.

[0184] Step 706: Process the initial pixel feature map using a classification network layer to obtain the initial object probability information of each pixel relative to the target object in multiple pixels of the image to be detected.

[0185] Step 708: Process the initial pixel feature map using a regression network layer to obtain the initial position information of each pixel relative to its corresponding target object detection region in the image to be detected; process the initial position information using a convolutional layer to obtain the position information of each pixel relative to its corresponding target object detection region in the image sample.

[0186] Step 710: Use the classification adjustment network layer to process the initial object probability information to obtain the first intermediate predicted object probability information.

[0187] Step 712: Use the feature compensation network layer to process the initial object probability information and initial position information to obtain the second intermediate predicted object probability information.

[0188] Step 714: Based on the first intermediate predicted object probability information and the second intermediate predicted object probability information, determine the object probability information of each pixel in the multiple pixels of the image sample relative to the target object.

[0189] In summary, the above method processes the image to be detected using an object detection model, obtaining the object probability information of each pixel in the image relative to the target object. This allows for the determination of the probability that each pixel in the image is a target object. Furthermore, it can obtain the positional information of each pixel relative to its corresponding target object detection region. By combining the object probability information and the positional information, the determination of the target object detection region not only considers the probability of a pixel being a target object but also obtains the positional information of each pixel relative to its corresponding target object detection region. This makes the determined target object detection region more accurate, thereby improving the accuracy of the detection results.

[0190] Corresponding to the above method embodiments, this specification also provides embodiments of object detection devices. Figure 8 A schematic diagram of an object detection device according to one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0191] The first determining module 802 is configured to determine the image to be detected that contains the target object;

[0192] The input module 804 is configured to input the image to be detected into the object detection model to obtain object probability information of each pixel relative to the target object and position information of each pixel relative to its corresponding target object detection region in a plurality of pixels of the image to be detected.

[0193] The second determining module 806 is configured to determine the target object detection region of the target object in the image to be detected based on the object probability information and the position information.

[0194] In an optional embodiment, the second determining module 806 is further configured to:

[0195] Based on the object probability information of each pixel relative to the target object, the target pixel corresponding to the target object is determined;

[0196] Based on the position information of each pixel relative to its corresponding target object detection area, the target position information of the target pixel relative to its corresponding target object detection area is determined.

[0197] Based on the target location information, the target object detection region is determined in the image to be detected.

[0198] In an optional embodiment, the device further includes a training module configured to:

[0199] Identify image samples containing the target object, and determine the target object detection region label in the image samples;

[0200] Based on the target object detection region label, determine the object probability information label of each pixel relative to the target object and the position information label of each pixel relative to its corresponding target object detection region label in the multiple pixels of the image sample;

[0201] The image sample is input into the object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, which are output by the object detection model.

[0202] The object detection model is trained based on the predicted object probability information and the object probability information label, the predicted location information and the location information label, until an object detection model that meets the training stopping condition is obtained.

[0203] In an optional embodiment, the training module is further configured to:

[0204] The image sample is input into an object detection model. In the object detection model, a feature extraction layer is used to extract features from the image sample to obtain an initial pixel feature map, wherein the initial pixel feature map includes multiple pixels of the image sample.

[0205] The initial pixel feature map is processed using a regression network layer to obtain the initial predicted position information of each pixel relative to its corresponding target object detection region among multiple pixels of the image sample.

[0206] The initial predicted position information is processed using a convolutional layer to obtain the predicted position information of each pixel relative to its corresponding target object detection region among multiple pixels of the image sample.

[0207] The initial pixel feature map is processed using a classification network layer to obtain the initial predicted object probability information of each pixel relative to the target object in multiple pixels of the image sample.

[0208] Based on the initial predicted location information and the initial predicted object probability information, the predicted object probability information of each pixel in the image sample relative to the target object is determined.

[0209] In an optional embodiment, the training module is further configured to:

[0210] The probability information of the initial predicted object is processed by the classification adjustment network layer to obtain the probability information of the first intermediate predicted object;

[0211] The initial predicted object probability information and the initial predicted location information are processed using a feature compensation network layer to obtain the second intermediate predicted object probability information;

[0212] Based on the first intermediate predicted object probability information and the second intermediate predicted object probability information, the predicted object probability information of each pixel in the image sample relative to the target object is determined.

[0213] In an optional embodiment, the classification adjustment network layer includes n classification adjustment units; the training module is further configured to:

[0214] The initial predicted object probability information is input into the i-th classification adjustment unit to obtain the i-th predicted object probability information output by the i-th classification adjustment unit, where the initial value of i is 1, i∈[1,n], and n is a positive integer;

[0215] The probability information of the i-th predicted object is input into the (i+1)-th classification adjustment unit to obtain the probability information of the (i+1)-th predicted object output by the (i+1)-th classification adjustment unit.

[0216] Determine if i+1 is greater than or equal to n;

[0217] If not, increment i by 1 and continue to execute the step of inputting the probability information of the i-th predicted object into the (i+1)-th classification adjustment unit;

[0218] If so, the probability information of the (i+1)th predicted object is determined as the probability information of the first intermediate predicted object.

[0219] In an optional embodiment, the training module is further configured to:

[0220] According to a preset label allocation strategy, intermediate object probability information labels are determined for each classification adjustment unit, wherein the preset label allocation strategy is to decrease the number of positive samples included in the intermediate object probability information labels allocated to each classification adjustment unit in turn.

[0221] Each classification adjustment unit is trained based on the predicted object probability information output by each classification adjustment unit and the intermediate object probability information label of each classification adjustment unit.

[0222] In an optional embodiment, the training module is further configured to:

[0223] The feature compensation network layer is used to concatenate the initial predicted object probability information and the initial predicted location information to obtain the first predicted feature.

[0224] The feature compensation network layer is used to concatenate and enhance the initial predicted object probability information and the initial predicted location information to obtain the second predicted feature.

[0225] The feature compensation network layer is used to process the initial predicted object probability information and the initial predicted location information to obtain a third predicted feature;

[0226] Based on the first prediction feature, the second prediction feature, and the third prediction feature, the probability information of the second intermediate prediction object is determined.

[0227] In an optional embodiment, the training module is further configured to:

[0228] The feature compensation network layer is used to concatenate the initial predicted object probability information and the initial predicted location information to obtain the first concatenated feature.

[0229] The initial predicted position information is inverted, and the initial predicted position information and the initial predicted object probability information obtained after inversion are concatenated and convolved to obtain the second concatenated feature.

[0230] The first and second spliced ​​features are subjected to feature enhancement processing to obtain the third predicted feature.

[0231] In an optional embodiment, the training module is further configured to:

[0232] Using a preset activation function, the first prediction feature, the second prediction feature, and the third prediction feature are normalized and convolved to obtain the probability information of the second intermediate prediction object.

[0233] In summary, the aforementioned device processes the image to be detected using an object detection model, enabling it to obtain the object probability information of each pixel in the image relative to the target object. This allows it to determine the probability that each pixel in the image is a target object. Furthermore, it can obtain the positional information of each pixel relative to its corresponding target object detection region. By combining the object probability information and the positional information, the determination of the target object detection region not only considers the probability of a pixel being a target object but also obtains the positional information of each pixel relative to its corresponding target object detection region. This makes the determined target object detection region more accurate, thereby improving the accuracy of the detection results.

[0234] The above is a schematic scheme of an object detection device according to this embodiment. It should be noted that the technical solution of this object detection device and the technical solution of the object detection method described above belong to the same concept. For details not described in detail in the technical solution of the object detection device, please refer to the description of the technical solution of the object detection method described above.

[0235] Corresponding to the above method embodiments, Figure 9 A flowchart of a training method for an object detection model according to an embodiment of this specification is shown, and the specific steps are described below.

[0236] Step 902: Determine an image sample containing the target object, and determine the target object detection region label in the image sample;

[0237] Step 904: Based on the target object detection region label, determine the object probability information label of each pixel relative to the target object and the position information label of each pixel relative to its corresponding target object detection region label in the multiple pixels of the image sample;

[0238] Step 906: Input the image sample into the object detection model to obtain the predicted object probability information of each pixel in the image sample relative to the target object, and the predicted position information of each pixel relative to its corresponding target object detection region, output by the object detection model.

[0239] Step 908: Train the object detection model based on the predicted object probability information and the object probability information label, the predicted location information and the location information label, until an object detection model that meets the training stopping condition is obtained.

[0240] Specifically, the training process of this object detection model is the same as that of the object detection model in the aforementioned object detection methods, and will not be repeated here.

[0241] Corresponding to the above method embodiments, this specification also provides embodiments of a training device for an object detection model. Figure 10 A schematic diagram of a training apparatus for an object detection model according to one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0242] The first determining module 1002 is configured to determine an image sample containing a target object, and to determine a target object detection region label of the target object in the image sample;

[0243] The second determining module 1004 is configured to determine, based on the target object detection region label, the object probability information label of each pixel relative to the target object and the position information label of each pixel relative to its corresponding target object detection region label in a plurality of pixels of the image sample;

[0244] The input module 1006 is configured to input the image sample into an object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, which are output by the object detection model.

[0245] The training module 1008 is configured to train the object detection model based on the predicted object probability information and the object probability information label, the predicted location information and the location information label, until an object detection model that meets the training stopping condition is obtained.

[0246] The above is a schematic scheme of a training device for an object detection model according to this embodiment. It should be noted that the technical solution of this object detection model training device belongs to the same concept as the technical solution of the object detection method described above. For details not described in detail in the technical solution of the object detection model training device, please refer to the description of the technical solution of the object detection method described above.

[0247] Corresponding to the above method embodiments, Figure 11 A flowchart of a pedestrian detection method according to an embodiment of this specification is shown, applied to a cloud-based device, and the specific steps are as follows.

[0248] Step 1102: Receive a pedestrian detection request sent by the receiving end device, wherein the pedestrian detection request carries an image to be detected;

[0249] Step 1104: Input the image to be detected into the object detection model to obtain the object probability information of each pixel relative to the pedestrian and the position information of each pixel relative to its corresponding pedestrian detection area in the multiple pixels of the image to be detected.

[0250] Step 1106: Determine the pedestrian detection region in the image to be detected based on the object probability information and the location information;

[0251] Step 1108: Based on the pedestrian detection area, process the image to be detected to obtain the processed target image;

[0252] Step 1110: Send the target image to the end-side device.

[0253] Specifically, the object detection model provided in the embodiments of this specification can be applied to the field of pedestrian detection.

[0254] End-side devices can be camera devices deployed on vehicles, such as dashcams. End-side devices can also be camera devices deployed in areas such as factories, businesses, residential areas, and roads.

[0255] In practice, the camera device can send an image of a captured area to the edge device. The camera device can be deployed on a vehicle, such as a dashcam, or in factories, businesses, residential areas, roads, etc. After receiving the image, the edge device can send a pedestrian detection request to the cloud device, carrying the image to be detected. The cloud device can input the image to be detected into an object detection model to determine the pedestrian detection region within the image. Based on this pedestrian detection region, the cloud device processes the image, marking the pedestrian detection region (i.e., pedestrian detection box) at the pedestrian's location in the processed target image. This target image is then sent to the edge device for rendering and display.

[0256] In summary, the above method processes the image to be detected using an object detection model, obtaining the object probability information of each pixel in the image relative to the target object. This allows for the determination of the probability that each pixel in the image is a target object. Furthermore, it can obtain the positional information of each pixel relative to its corresponding target object detection region. By combining the object probability information and the positional information, the determination of the target object detection region not only considers the probability of a pixel being a target object but also obtains the positional information of each pixel relative to its corresponding target object detection region. This makes the determined target object detection region more accurate, thereby improving the accuracy of the detection results.

[0257] Corresponding to the above method embodiments, this specification also provides embodiments of pedestrian detection devices. Figure 12 A schematic diagram of a pedestrian detection device according to one embodiment of this specification is shown. Figure 12 As shown, the device includes:

[0258] The receiving module 1202 is configured to receive a pedestrian detection request sent by the end-side device, wherein the pedestrian detection request carries an image to be detected;

[0259] The input module 1204 is configured to input the image to be detected into the object detection model to obtain object probability information of each pixel relative to the pedestrian and position information of each pixel relative to its corresponding pedestrian detection area in the multiple pixels of the image to be detected.

[0260] The determination module 1206 is configured to determine the pedestrian detection region of the pedestrian in the image to be detected based on the object probability information and the location information.

[0261] The processing module 1208 is configured to process the image to be detected based on the pedestrian detection area to obtain the processed target image;

[0262] The sending module 1210 is configured to send the target image to the end-side device.

[0263] In summary, the aforementioned device processes the image to be detected using an object detection model, enabling it to obtain the object probability information of each pixel in the image relative to the target object. This allows it to determine the probability that each pixel in the image is a target object. Furthermore, it can obtain the positional information of each pixel relative to its corresponding target object detection region. By combining the object probability information and the positional information, the determination of the target object detection region not only considers the probability of a pixel being a target object but also obtains the positional information of each pixel relative to its corresponding target object detection region. This makes the determined target object detection region more accurate, thereby improving the accuracy of the detection results.

[0264] The above is a schematic scheme of a pedestrian detection device according to this embodiment. It should be noted that the technical solution of this pedestrian detection device and the technical solution of the pedestrian detection method described above belong to the same concept. For details not described in detail in the technical solution of the pedestrian detection device, please refer to the description of the technical solution of the pedestrian detection method described above.

[0265] Corresponding to the above method embodiments, Figure 13 A flowchart of another pedestrian detection method according to an embodiment of this specification is shown, applied to a cloud-based device, and the specific steps are as follows.

[0266] Step 1302: Receive road images captured by the vehicle's camera device during the vehicle's movement;

[0267] Step 1304: Input the road image into the object detection model to obtain the object probability information of each pixel relative to the pedestrian in multiple pixels of the road image, and the position information of each pixel relative to its corresponding pedestrian detection area;

[0268] Step 1306: Determine the pedestrian detection region of the pedestrian in the road image based on the object probability information and the location information;

[0269] Step 1308: Determine the vehicle's trajectory based on the pedestrian detection area;

[0270] Step 1310: Control the vehicle according to the driving trajectory.

[0271] Specifically, the object detection model provided in the embodiments of this specification can also be used in the field of autonomous driving. In specific implementation, this object detection model can be used to determine the pedestrian detection area included in the road image captured by the vehicle's camera device, thereby realizing the localization of pedestrians in front of the vehicle. This facilitates subsequent planning of the vehicle's driving trajectory based on the pedestrian detection area and controlling the vehicle's driving according to the driving trajectory, ensuring the safety of the vehicle during autonomous driving.

[0272] Corresponding to the above method embodiments, this specification also provides embodiments of pedestrian detection devices. Figure 14 A schematic diagram of another pedestrian detection device provided in one embodiment of this specification is shown. Figure 14 As shown, the device includes:

[0273] The receiving module 1402 is configured to receive road images captured by the vehicle's camera device during the vehicle's movement.

[0274] The input module 1404 is configured to input the road image into the object detection model to obtain object probability information of each pixel relative to a pedestrian and position information of each pixel relative to its corresponding pedestrian detection area in the multiple pixels of the road image.

[0275] The first determining module 1406 is configured to determine the pedestrian detection region of the pedestrian in the road image based on the object probability information and the location information.

[0276] The second determining module 1408 is configured to determine the vehicle's trajectory based on the pedestrian detection area;

[0277] The control module 1410 is configured to control the vehicle according to the driving trajectory.

[0278] The above is a schematic scheme of a pedestrian detection device according to this embodiment. It should be noted that the technical solution of this pedestrian detection device and the technical solution of the pedestrian detection method described above belong to the same concept. For details not described in detail in the technical solution of the pedestrian detection device, please refer to the description of the technical solution of the pedestrian detection method described above.

[0279] Figure 15 A structural block diagram of a computing device 1500 according to one embodiment of this specification is shown. The components of the computing device 1500 include, but are not limited to, a memory 1510 and a processor 1520. The processor 1520 is connected to the memory 1510 via a bus 1530, and a database 1550 is used to store data.

[0280] The computing device 1500 also includes an access device 1540, which enables the computing device 1500 to communicate via one or more networks 1560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0281] In one embodiment of this application, the aforementioned components of the computing device 1500 and Figure 15 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 15 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0282] The computing device 1500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1500 can also be a mobile or stationary server.

[0283] The processor 1520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0284] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the above method.

[0285] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0286] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the method described above belong to the same concept, and all details not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the method described above.

[0287] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method.

[0288] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the method described above.

[0289] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0290] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0291] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0292] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0293] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An object detection method, comprising: Identify the image to be detected that contains the target object; The image to be detected is input into an object detection model to obtain object probability information of each pixel relative to the target object and position information of each pixel relative to its corresponding target object detection region. The object detection model includes a classification adjustment network layer and a feature compensation network layer. The classification adjustment network layer is used to adjust the initial predicted object probability information, and the feature compensation network layer is used to perform feature compensation on the initial predicted object probability information based on the initial predicted position information. The object probability information is determined based on the initial predicted position information and the initial predicted object probability information. Based on the object probability information and the location information, the target object detection region of the target object is determined in the image to be detected.

2. The method according to claim 1, wherein determining the target object detection region in the image to be detected based on the object probability information and the location information comprises: Based on the object probability information of each pixel relative to the target object, the target pixel corresponding to the target object is determined; Based on the position information of each pixel relative to its corresponding target object detection area, the target position information of the target pixel relative to its corresponding target object detection area is determined. Based on the target location information, the target object detection region is determined in the image to be detected.

3. The method according to claim 1, further comprising, before inputting the image to be detected into the object detection model: Identify image samples containing the target object, and determine the target object detection region label in the image samples; Based on the target object detection region label, determine the object probability information label of each pixel relative to the target object and the position information label of each pixel relative to its corresponding target object detection region label in the multiple pixels of the image sample; The image sample is input into the object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, which are output by the object detection model. The object detection model is trained based on the predicted object probability information and the object probability information label, the predicted location information and the location information label, until an object detection model that meets the training stopping condition is obtained.

4. The method according to claim 3, wherein inputting the image sample into an object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, output by the object detection model, comprises: The image sample is input into an object detection model. In the object detection model, a feature extraction layer is used to extract features from the image sample to obtain an initial pixel feature map, wherein the initial pixel feature map includes multiple pixels of the image sample. The initial pixel feature map is processed using a regression network layer to obtain the initial predicted position information of each pixel relative to its corresponding target object detection region among multiple pixels of the image sample. The initial predicted position information is processed using a convolutional layer to obtain the predicted position information of each pixel relative to its corresponding target object detection region among multiple pixels of the image sample. The initial pixel feature map is processed using a classification network layer to obtain the initial predicted object probability information of each pixel relative to the target object in multiple pixels of the image sample. Based on the initial predicted location information and the initial predicted object probability information, the predicted object probability information of each pixel in the image sample relative to the target object is determined.

5. The method according to claim 4, wherein determining the predicted object probability information of each pixel relative to the target object among multiple pixels of the image sample based on the initial predicted location information and the initial predicted object probability information comprises: The probability information of the initial predicted object is processed by the classification adjustment network layer to obtain the probability information of the first intermediate predicted object; The initial predicted object probability information and the initial predicted location information are processed using a feature compensation network layer to obtain the second intermediate predicted object probability information; Based on the first intermediate predicted object probability information and the second intermediate predicted object probability information, the predicted object probability information of each pixel in the image sample relative to the target object is determined.

6. The method according to claim 5, wherein the classification adjustment network layer comprises n classification adjustment units; Accordingly, the step of inputting the initial predicted object probability information into the classification adjustment network layer and processing the initial predicted object probability information using the classification adjustment network layer to obtain the first intermediate predicted object probability information includes: The initial predicted object probability information is input into the i-th classification adjustment unit to obtain the i-th predicted object probability information output by the i-th classification adjustment unit, where the initial value of i is 1, i∈[1,n], and n is a positive integer; The probability information of the i-th predicted object is input into the (i+1)-th classification adjustment unit to obtain the probability information of the (i+1)-th predicted object output by the (i+1)-th classification adjustment unit. Determine if i+1 is greater than or equal to n; If not, increment i by 1 and continue to execute the step of inputting the probability information of the i-th predicted object into the (i+1)-th classification adjustment unit; If so, the probability information of the (i+1)th predicted object is determined as the probability information of the first intermediate predicted object.

7. The method according to claim 6, further comprising: According to a preset label allocation strategy, intermediate object probability information labels are determined for each classification adjustment unit, wherein the preset label allocation strategy is to decrease the number of positive samples included in the intermediate object probability information labels allocated to each classification adjustment unit in turn. Each classification adjustment unit is trained based on the predicted object probability information output by each classification adjustment unit and the intermediate object probability information label of each classification adjustment unit.

8. The method according to claim 5, wherein processing the initial predicted object probability information and the initial predicted position information using the feature compensation network layer to obtain the second intermediate predicted object probability information includes: The feature compensation network layer is used to concatenate the initial predicted object probability information and the initial predicted location information to obtain the first predicted feature. The feature compensation network layer is used to concatenate and enhance the initial predicted object probability information and the initial predicted location information to obtain the second predicted feature. The feature compensation network layer is used to process the initial predicted object probability information and the initial predicted location information to obtain a third predicted feature; Based on the first prediction feature, the second prediction feature, and the third prediction feature, the probability information of the second intermediate prediction object is determined.

9. The method according to claim 8, wherein processing the initial predicted object probability information and the initial predicted location information using the feature compensation network layer to obtain the third predicted feature includes: The feature compensation network layer is used to concatenate the initial predicted object probability information and the initial predicted location information to obtain the first concatenated feature. The initial predicted position information is inverted, and the initial predicted position information and the initial predicted object probability information obtained after inversion are concatenated and convolved to obtain the second concatenated feature. The first and second spliced ​​features are subjected to feature enhancement processing to obtain the third predicted feature.

10. The method according to claim 8, wherein determining the probability information of the second intermediate prediction object based on the first prediction feature, the second prediction feature, and the third prediction feature comprises: Using a preset activation function, the first prediction feature, the second prediction feature, and the third prediction feature are normalized and convolved to obtain the probability information of the second intermediate prediction object.

11. A method for training an object detection model, comprising: Identify image samples containing the target object, and determine the target object detection region label in the image samples; Based on the target object detection region label, determine the object probability information label of each pixel relative to the target object and the position information label of each pixel relative to its corresponding target object detection region label in the multiple pixels of the image sample; The image sample is input into an object detection model to obtain the predicted object probability information of each pixel relative to the target object and the predicted position information of each pixel relative to its corresponding target object detection region, which are output by the object detection model. The object detection model includes a classification adjustment network layer and a feature compensation network layer. The classification adjustment network layer is used to adjust the initial predicted object probability information, and the feature compensation network layer is used to perform feature compensation on the initial predicted object probability information based on the initial predicted position information. The predicted object probability information is determined based on the initial predicted position information and the initial predicted object probability information. The object detection model is trained based on the predicted object probability information and the object probability information label, the predicted location information and the location information label, until an object detection model that meets the training stopping condition is obtained.

12. A pedestrian detection method, applied to cloud-based devices, comprising: A pedestrian detection request sent by a receiving end-side device, wherein the pedestrian detection request carries an image to be detected; The image to be detected is input into an object detection model to obtain object probability information of each pixel relative to a pedestrian and position information of each pixel relative to its corresponding pedestrian detection region. The object detection model includes a classification adjustment network layer and a feature compensation network layer. The classification adjustment network layer is used to adjust the initial predicted object probability information, and the feature compensation network layer is used to perform feature compensation on the initial predicted object probability information based on the initial predicted position information. The object probability information is determined based on the initial predicted position information and the initial predicted object probability information. Based on the object probability information and the location information, determine the pedestrian detection region of the pedestrian in the image to be detected; Based on the pedestrian detection area, the image to be detected is processed to obtain the processed target image; The target image is sent to the end-side device.

13. A pedestrian detection method, applied to cloud-based devices, comprising: The vehicle's camera receives road images captured by the vehicle's camera during its movement. The road image is input into an object detection model to obtain object probability information of each pixel relative to a pedestrian and position information of each pixel relative to its corresponding pedestrian detection area. The object detection model includes a classification adjustment network layer and a feature compensation network layer. The classification adjustment network layer is used to adjust the initial predicted object probability information, and the feature compensation network layer is used to perform feature compensation on the initial predicted object probability information based on the initial predicted position information. The object probability information is determined based on the initial predicted position information and the initial predicted object probability information. Based on the object probability information and the location information, determine the pedestrian detection area of ​​the pedestrian in the road image; The vehicle's trajectory is determined based on the pedestrian detection area; The vehicle is controlled according to the driving trajectory.

14. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

Images