Pedestrian ranging method, device and readable storage medium

Abstract

Embodiments of the present application provide a pedestrian ranging method, device and readable storage medium. The method comprises: acquiring a vehicle front view image collected by a monocular camera; inputting the vehicle front view image into a pre-trained first deep learning model for semantic segmentation processing and image depth estimation processing, to obtain semantic information and depth information of the vehicle front view image; cropping a target image of a pedestrian from the vehicle front view image using the semantic information; calculating a first distance between a pixel point in the target image and the monocular camera according to the depth information; calculating a second distance between the pedestrian and the monocular camera; and performing weighted summation on the first distance and the second distance to obtain a target distance between the pedestrian and the vehicle. Embodiments of the present application can effectively improve the accuracy of pedestrian ranging results.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a pedestrian ranging method, apparatus, and readable storage medium. Background Technology

[0002] In recent years, deep learning has demonstrated unique advantages in fields such as Artificial Intelligence & Internet of Things (AIOT) and autonomous driving. In autonomous driving or security equipment, the perception of targets ahead, especially pedestrians and vehicles, is crucial for subsequent distance measurement and judgment, significantly impacting the real-time performance of vehicle control or security alarms.

[0003] Currently, monocular depth ranging based on vision sensors is the mainstream approach in existing ranging technologies. This method uses only a single two-dimensional image for distance measurement, making it highly dependent on camera calibration. If encountering inclines or declines, correction of the vanishing point is also required. Furthermore, the distance measurement results will differ for pedestrians in different postures, resulting in inaccurate ranging results. Summary of the Invention

[0004] This invention provides a pedestrian distance measurement method, apparatus, and readable storage medium, which can effectively improve the accuracy of pedestrian distance measurement results.

[0005] In a first aspect, embodiments of the present invention disclose a pedestrian distance measurement method, the method comprising:

[0006] Acquire the front view image of the vehicle captured by a monocular camera;

[0007] The vehicle front view image is input into a pre-trained first deep learning model for semantic segmentation and image depth estimation to obtain the semantic and depth information of the vehicle front view image.

[0008] The semantic information is used to crop the target image of the pedestrian from the vehicle front view image;

[0009] Calculate the first distance between the pixels in the target image and the monocular camera based on the depth information;

[0010] Calculate the second distance between the pedestrian and the monocular camera;

[0011] The first distance and the second distance are weighted and summed to obtain the target distance between the pedestrian and the vehicle.

[0012] Secondly, embodiments of the present invention disclose a pedestrian ranging device, the device comprising:

[0013] The image acquisition module is used to acquire the front view image of the vehicle captured by the monocular camera;

[0014] The image processing module is used to input the vehicle front view image into a pre-trained first deep learning model for semantic segmentation and image depth estimation to obtain the semantic information and depth information of the vehicle front view image.

[0015] An image cropping module is used to crop a target image of a pedestrian from the vehicle front view image using the semantic information;

[0016] The first distance calculation module is used to calculate the first distance between the pixels in the target image and the monocular camera based on the depth information.

[0017] The second distance calculation module is used to calculate the second distance between the pedestrian and the monocular camera;

[0018] The target distance calculation module is used to perform a weighted summation of the first distance and the second distance to obtain the target distance between the pedestrian and the vehicle.

[0019] Thirdly, embodiments of the present invention disclose a machine-readable storage medium having instructions stored thereon that, when executed by one or more processors of the device, cause the device to perform the pedestrian ranging method as described in the first aspect above.

[0020] The embodiments of the present invention have the following advantages:

[0021] The pedestrian ranging method provided in this invention performs image processing on a vehicle front view image using a first deep learning model to obtain semantic and depth information of the vehicle front view image. Then, based on the semantic information, a target image of the pedestrian is cropped from the vehicle front view image. The first distance is calculated using the depth information of the pixels in the cropped target image. This effectively reduces the influence of the background area in the vehicle front view image on the ranging result, and the depth information is more robust. Furthermore, the second distance between the pedestrian and the monocular camera is calibrated using the first distance calculated based on the depth information, which can effectively improve the accuracy of the pedestrian ranging result. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the steps of an embodiment of the pedestrian distance measurement method of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of a first deep learning model according to the present invention;

[0025] Figure 3 This is a schematic diagram of the installation position of a monocular camera and radar according to the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a second deep learning model according to the present invention;

[0027] Figure 5 This is a schematic diagram of a depth region capture according to the present invention;

[0028] Figure 6 This is a structural block diagram of an embodiment of a pedestrian ranging device according to the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0031] Reference Figure 1 The diagram illustrates a step flowchart of an embodiment of the pedestrian distance measurement method of the present invention, which may include the following steps:

[0032] Step 101: Obtain the front view image of the vehicle captured by the monocular camera.

[0033] Step 102: Input the vehicle front view image into a pre-trained first deep learning model for semantic segmentation and image depth estimation to obtain the semantic and depth information of the vehicle front view image.

[0034] Step 103: Use the semantic information to crop the target image of the pedestrian from the vehicle front view image.

[0035] Step 104: Calculate the first distance between the pixels in the target image and the monocular camera based on the depth information.

[0036] Step 105: Calculate the second distance between the pedestrian and the monocular camera.

[0037] Step 106: Perform a weighted summation of the first distance and the second distance to obtain the target distance between the pedestrian and the vehicle.

[0038] The pedestrian ranging method provided in this invention can be applied to edge computing devices. It should be noted that edge computing refers to providing the nearest service by utilizing network, computing, storage, and application capabilities on the side closest to the object or data source. In a sense, edge computing can be seen as a supplement to cloud computing or a preprocessing step of cloud computing. The edge computing device is used to provide edge computing services. In this invention embodiment, the edge computing device can include two types: one type requires connection to an external camera, and the other type integrates a camera. A target detection system can be constructed based on at least one of these two types of edge computing devices. For example, the edge computing device can receive images to be detected from one or more terminal devices; or, the edge computing device can acquire images to be detected through its integrated camera. The edge computing device in this invention embodiment can be a device for implementing edge computing functions, such as an independent device like a server or network element, or multiple devices that jointly implement edge computing functions.

[0039] The edge computing device can be deployed in multiple locations in the network, such as on the macro base station (eNode B) side, radio network controller (RNC) side, multi-radio access technology (multi-RAN) cellular convergence point side, or core network edge of an LTE (Long Term Evolution) network. This embodiment of the invention does not limit the specific deployment of the edge computing device.

[0040] It is understood that the edge computing device operates at the network edge and is logically independent of other parts of the network, making it suitable for applications requiring high security. Edge computing devices typically have high computing power, thus making them suitable for analyzing large amounts of data, such as executing the pedestrian ranging method in this invention to analyze a vehicle's forward view image and determine the target distance between a pedestrian and a vehicle.

[0041] In this embodiment of the invention, a monocular camera can be used to capture a frontal view image of a vehicle, and the monocular camera can be installed at the front of the vehicle. The monocular camera establishes a communication connection with an edge computing device through a wired or wireless network, and transmits the captured frontal view image of the vehicle to the edge computing device.

[0042] After acquiring the vehicle's forward-view image, the edge computing device performs image analysis processing to determine the target distance between pedestrians and vehicles within the image. Specifically, the vehicle's forward-view image is first input into a pre-trained first deep learning model for semantic segmentation and image depth estimation to obtain semantic and depth information of the vehicle's forward-view image. It is understood that the first deep learning model in this embodiment possesses semantic segmentation and image depth estimation capabilities for the input image. The first deep learning model can be any neural network model in the art that excels at semantic segmentation and depth estimation of images. For example, the first deep learning model can be a UNet model, a DeepLabv3 model, etc.

[0043] The semantic information reflects the category information of each pixel in the vehicle's front view image, distinguishing whether a pixel belongs to the foreground or background image, and providing the outline information of objects in the foreground image, such as vehicles, pedestrians, and traffic lights. The depth information reflects the distance information between each pixel in the vehicle's front view image and the monocular camera. As an example, the semantic information and the depth information can be label maps. The label map corresponding to the semantic information is marked with the category labels corresponding to each pixel in the vehicle's front view image, and the label map corresponding to the depth information can be a depth map, where the distance between the pixel in the vehicle's front view image and the monocular camera is used as the pixel value. Alternatively, the semantic information and the depth information can be matrices, with the matrix dimensions matching the dimensions of the vehicle's front view image. Each value in the matrix corresponding to the semantic information represents the category of the corresponding pixel in the vehicle's front view image. For example, the values ​​in the semantic information matrix can include 0, 5, and 2, where "0" represents the background, "5" represents a vehicle, "2" represents a pedestrian, and so on. Each value in the matrix corresponding to the depth information represents the distance between the corresponding pixel in the vehicle's front view image and the monocular camera.

[0044] Based on semantic information, a pedestrian target image can be cropped from a vehicle's front view image. For example, pixels categorized as "pedestrian" can be extracted from the vehicle's front view image based on the category of each pixel reflected in the semantic information, thus obtaining the pedestrian target image. For instance, the top-left pixel of each "pedestrian" region reflected in the semantic information can be taken, and then a four-neighbor nearest neighbor search can be used to determine other surrounding pixels reflecting "pedestrian," thereby determining a suitable pedestrian bounding box. If the number of pixels reflecting "pedestrian" within the region after the nearest neighbor search is less than a preset threshold, such as 20, that region can be considered noise and discarded. After the search, the coordinates of the pedestrian target bounding box can be obtained by taking xmin, xmax, ymin, and ymax within the region to construct four points: top-left, top-right, bottom-left, and bottom-right. Based on the pedestrian target bounding box, the pedestrian target image can be segmented from the vehicle's front view image.

[0045] Next, the first distance between pixels in the target image and the monocular camera is calculated based on the depth information of the vehicle's front view image. It should be noted that this first distance reflects the combined distance between all pixels in the target image and the monocular camera, not the distance between a single pixel and the monocular camera. For example, each pixel in the target image is mapped into a three-dimensional space based on its distance to the monocular camera. The pixels in the target image form a sphere, and the distance between the center of this sphere and the monocular camera can be used as the first distance in this embodiment of the invention. Alternatively, the average value, median value, etc., of the distances between each pixel in the target image and the monocular camera can be calculated, and the calculation result can be used as the first distance in this embodiment of the invention, and so on.

[0046] The second distance in this invention reflects the relative distance between the pedestrian and the monocular camera, which can be calculated based on the principle of "pinhole imaging". Specifically, assuming the focal length of the monocular camera is F, the reference height of the pedestrian is H, and the pixel height of the pedestrian is h, that is, the pixel distance between the highest and lowest points of the pixels representing the pedestrian in the vehicle's front view image is h, then the second distance d2 can be expressed as:

[0047] d2=(F*H) / h (1)

[0048] After calculating the first distance and the second distance, a weighted sum of the first distance and the second distance is performed to obtain the target distance between the pedestrian and the vehicle. For example, in this embodiment of the invention, the target distance between the pedestrian and the vehicle can be expressed as: d = d1*w1 + d2*w2, where d1 is the first distance between a pixel in the target image and the monocular camera, w1 is the weight of the first distance, d2 is the second distance between the pedestrian and the monocular camera, and w2 is the weight of the second distance. w1 and w2 can be set according to actual needs; for example, w1 = w2 = 0.5.

[0049] It should be noted that before calculating the second distance between the pedestrian and the monocular camera, camera calibration can be performed first to improve the accuracy of the calculation results. Camera calibration includes determining the camera's intrinsic and extrinsic parameters. Intrinsic parameters include the camera's field of view, focal length, and distortion coefficient, while extrinsic parameters include the camera's mounting height and distance from the center of the vehicle's rear axle.

[0050] The pedestrian ranging method provided in this invention combines semantic and depth information of a vehicle's front view image for ranging. Specifically, a target image of a pedestrian is cropped from the vehicle's front view image based on semantic information, and the distance is calculated using the depth information of the pixels in the cropped target image. This effectively reduces the influence of the background area in the vehicle's front view image on the ranging result, and the depth information is more robust. The second distance is calibrated using the first distance calculated based on the depth information, which can effectively improve the accuracy of the ranging result.

[0051] In an optional embodiment of the present invention, the first deep learning model includes a semantic segmentation branch and a depth estimation branch, wherein the semantic segmentation branch and the depth estimation branch reuse a feature extraction network and a feature aggregation network, the semantic segmentation branch further includes a category prediction network, and the depth estimation branch further includes a depth prediction network.

[0052] Reference Figure 2 The diagram illustrates the structure of a first deep learning model provided in an embodiment of the present invention. Figure 2 As shown, the first deep learning model 20 includes a feature extraction network 201 (Backbone), a feature aggregation network 202 (Neck), a category prediction network 203 (Head1), and a depth prediction network 204 (Head2). The feature extraction network 201 is mainly used to extract image features; the feature aggregation network 202 is used to aggregate the image features extracted by the feature extraction network 201; the category prediction network 203 is used to classify and discriminate the aggregated features to determine the category of each pixel; and the depth prediction network 204 is used to obtain the depth information of the image, that is, the distance information between each pixel in the image and the monocular camera.

[0053] Semantic segmentation and image depth estimation are both pixel-level classification tasks. The processing of feature maps in Backbone is similar, and since both tasks are pixel-level classification tasks at different scales, the extracted features can be reused to some extent. In some embodiments, the losses from both tasks can even be fused to improve the processing performance of semantic segmentation and depth estimation. In this embodiment, feature extraction network 201, feature aggregation network 202, and category prediction network 203 constitute the semantic segmentation branch, while feature extraction network 201, feature aggregation network 202, and depth prediction network 204 constitute the depth estimation branch. It can be seen that the semantic segmentation branch and the depth estimation branch reuse feature extraction network 201 and feature aggregation network 202. Compared to using independent semantic segmentation and depth estimation models to perform semantic segmentation and image depth estimation on vehicle front-view images respectively, this embodiment utilizes the similarity between the segmentation model and the depth estimation model on the depth map, employing a first deep learning model with a reused network structure to simultaneously perform semantic segmentation and depth estimation on vehicle front-view images. This effectively reduces the model parameters of the deep learning model and saves storage resources on edge computing devices.

[0054] Optionally, before inputting the vehicle front view image into a pre-trained first deep learning model for semantic segmentation and image depth estimation, the method further includes:

[0055] Step S11: Obtain a training sample set, which includes pedestrian sample images and semantic and depth labels of the pedestrian sample images;

[0056] Step S12: In each round of training, the pedestrian sample image is input into the first deep learning model for processing to obtain the semantic segmentation result and the first depth information output by the first deep learning model.

[0057] Step S13: Adjust the first depth information using the semantic segmentation result to obtain the second depth information;

[0058] Step S14: Calculate the first loss value of the semantic segmentation branch based on the semantic label and the semantic segmentation result, and calculate the second loss value of the depth estimation branch based on the depth label and the second depth information;

[0059] Step S15: Perform a weighted summation of the first loss value and the second loss value to obtain the joint loss value of the first deep learning model;

[0060] Step S16: Adjust the model parameters of the first deep learning model according to the joint loss value, and perform the next round of training until the joint loss value meets the preset conditions, and obtain the trained first deep learning model.

[0061] In this embodiment of the invention, when training the first deep learning model, the semantic segmentation branch and the depth estimation branch can be jointly trained.

[0062] Specifically, in each round of training, pedestrian sample images from the training sample set are input into a first deep learning model for processing to obtain semantic segmentation results and first depth information. The first depth information is then adjusted using the semantic segmentation results to obtain second depth information. For example, adjusting the first depth information using the semantic segmentation results to obtain the second depth information includes: randomly selecting N regions from the first depth information using the semantic segmentation results, and fine-tuning the depth values ​​of each pixel within these N regions. For example, a deep network (RCNN in Faster R-CNN) can be constructed primarily using the N random regions from the first depth information for cascaded training. When predicting depth information, the pedestrian regions in the first depth map determined by the segmentation results are directly used to estimate the depth of the pedestrian regions. The semantic segmentation results can reflect the category information of each pixel in the image, determining the region to which each object belongs in the image. Based on the region to which each object belongs, adjusting the depth values ​​of each pixel within that region can make the depth values ​​of individual targets in the final second depth information more accurate.

[0063] Next, the loss values ​​for the semantic segmentation branch and the depth estimation branch are calculated separately. Specifically, the first loss value for the semantic segmentation branch is calculated based on the semantic labels and semantic segmentation results of the pedestrian sample images; the second loss value for the depth estimation branch is calculated based on the depth labels and second depth information. It should be noted that the same or different loss functions can be used to calculate the first and second loss values. The loss functions may include, but are not limited to: the Softmax function, zero-one loss, absolute value loss, logarithmic loss, exponential loss, cross-entropy loss, etc.

[0064] The joint loss value of the first deep learning model can be obtained by weighted summing of the first loss value and the second loss value. In practical applications, the weights of the first and second loss values ​​can be set according to the performance requirements of semantic segmentation and image depth estimation processing on the network model. This embodiment of the invention does not impose specific limitations on these settings.

[0065] Finally, the model parameters of the first deep learning model are adjusted based on the joint loss value. For example, the joint loss value is backpropagated and the gradient of the first deep learning model is updated. The model parameters may include, but are not limited to, the weights and biases of each network layer in the first deep learning model.

[0066] If the joint loss value does not meet the preset conditions, for example, if the joint loss value is greater than the preset loss value threshold, then the next round of training is performed. If the joint loss value meets the preset conditions, for example, if the joint loss value is less than or equal to the preset loss value threshold, then training ends, and the first deep learning model that has been trained is obtained.

[0067] Optionally, step S11, obtaining the training sample set, includes:

[0068] Sub-step S111: Acquire pedestrian sample images using a monocular camera and acquire 3D data of the pedestrians using radar; the monocular camera and the radar are coplanar;

[0069] Sub-step S112: Analyze the category to which each pixel in the pedestrian sample image belongs to obtain the semantic label of the pedestrian sample image;

[0070] Sub-step S113: Generate depth labels corresponding to the pedestrian sample images based on the three-dimensional data;

[0071] Sub-step S114: Construct a training sample set based on the collected pedestrian sample images and the semantic and depth labels of the pedestrian sample images.

[0072] In this embodiment of the invention, training samples can be collected using a monocular camera and radar. It should be noted that the monocular camera and radar are coplanar. (Refer to...) Figure 3 This diagram illustrates the installation location of a monocular camera and radar according to an embodiment of the present invention. Figure 3 As shown, the ends of the monocular camera and radar are aligned, and the distance between them and the object to be photographed is the same. The monocular camera is used to acquire sample images of pedestrians, that is, two-dimensional planar data of pedestrians; the radar is used to acquire three-dimensional data of pedestrians.

[0073] After data collection, the category of each pixel in the pedestrian sample image is analyzed to obtain the semantic label of the pedestrian sample image. It can be understood that the semantic label is used to reflect the category to which each pixel in the pedestrian sample image belongs.

[0074] The 3D data acquired by radar, also known as point cloud data, contains 3D coordinate information for each point, corresponding to the X, Y, and Z coordinate elements. Sometimes, it also includes color information, reflection intensity information, and echo count information. Depth labels corresponding to pedestrian sample images can be generated based on the 3D data acquired by radar. For example, by extracting the Z-axis information of each point in the 3D data, the depth information of each pixel in the pedestrian sample image can be obtained, i.e., the depth label. This depth label reflects the distance between each pixel in the pedestrian sample image and the monocular camera.

[0075] Once pedestrian sample images, semantic labels, and depth labels are obtained, a training sample set can be constructed, and then the first deep learning model in this invention can be iteratively trained using the training sample set.

[0076] In an optional embodiment of the present invention, the depth information of the vehicle front view image includes a depth map corresponding to the vehicle front view image, and step 104, calculating the first distance between the pixels in the target image and the monocular camera based on the depth information, includes:

[0077] Step S21: Input the target image into the second deep learning model to perform pose prediction and orientation prediction, and obtain the pedestrian's pose information and orientation information;

[0078] Step S22: Based on the posture information and the orientation information, determine the depth region corresponding to the pedestrian in the depth map;

[0079] Step S23: Determine the first Gaussian curve in the horizontal direction and the second Gaussian curve in the vertical direction for each pixel in the depth region;

[0080] Step S24: Integrate the first Gaussian curve and the second Gaussian curve respectively to obtain the first depth value and the second depth value;

[0081] Step S25: Calculate the average of the first depth value and the second depth value to obtain the first distance.

[0082] The depth map is used to reflect the depth value of each pixel in the vehicle's front view image, that is, the distance between each pixel and the monocular camera.

[0083] In this embodiment of the invention, after cropping the target image of a pedestrian from the vehicle's front view image using semantic information, the target image can be further input into a second deep learning model for pose and orientation prediction to obtain the pedestrian's pose and orientation information. The second deep learning model can be any neural network model in the art that is adept at pose and orientation prediction. For example, referring to... Figure 4 The diagram illustrates the structure of a second deep learning model provided in an embodiment of the present invention. Figure 4 As shown, the second deep learning model 40 includes a feature extraction network 401, a pose prediction network 402, and an orientation prediction network 403. The feature extraction network 401 extracts image features; the pose prediction network 402 analyzes the extracted features to predict the pedestrian's pose, which can include any of the following: standing, sitting, bending over, lying down, etc. The orientation prediction network 403 analyzes the extracted features to predict the pedestrian's orientation, which can include any of the following: forward, backward, left, right, left-front, right-front, left-rear, right-rear.

[0084] Next, based on the pedestrian's posture and orientation information, the depth region corresponding to the pedestrian is determined in the depth map of the vehicle's front view image. This depth region reflects the depth information corresponding to the rigid region of the pedestrian. It is understood that pedestrians are non-rigid objects, and considering the influence of the background region in the vehicle's front view image on the depth map, the pedestrian's depth center region may not necessarily appear in the center of the depth map. Therefore, this embodiment of the invention utilizes the pedestrian's posture and orientation information to determine the pedestrian's prior depth center region in the depth map, i.e., the depth region in this embodiment, to reduce the influence of the background region on the pedestrian's corresponding depth information, thereby improving the accuracy of pedestrian distance measurement results.

[0085] Optionally, step S22, which involves determining the depth region corresponding to the pedestrian in the depth map based on the pose information and the orientation information, includes:

[0086] Sub-step S221: Determine the first candidate region to be cropped in the depth map based on the posture information and the preset first cropping strategy. The first cropping strategy is used to indicate the region to be cropped corresponding to each posture.

[0087] Sub-step S222: Determine the second candidate region to be cropped in the depth map based on the orientation information and the preset second cropping strategy, wherein the second cropping strategy is used to indicate the region to be cropped corresponding to each orientation;

[0088] Sub-step S223: Find the intersection of the first candidate region and the second candidate region to obtain the depth region corresponding to the pedestrian in the depth map.

[0089] In this embodiment of the invention, the corresponding region to be captured can be determined in advance for each posture or orientation. As an example, referring to Table 1, a first capture strategy provided by this embodiment of the invention is shown.

[0090] Table 1

[0091] Standing depth Figure 1 / 2 Height Zone Sitting depth Figure 1 The lower half of the area at a height of / 2 bend over depth Figure 1 The lower half of the area at a height of / 2 Lying down depth Figure 1 / 2 length region

[0092] Based on the pose information output by the second deep learning model and the first cropping strategy shown in Table 1, the first candidate region to be cropped corresponding to the pose information can be determined.

[0093] As another example, referring to Table 2, a second interception strategy provided by an embodiment of the present invention is shown.

[0094] Table 2

[0095]

[0096]

[0097] Based on the orientation information output by the second deep learning model and the second cropping strategy shown in Table 2, the second candidate region to be cropped corresponding to the orientation information can be determined.

[0098] Finding the intersection of the first and second candidate regions yields the depth region corresponding to the pedestrian in the depth map. (Refer to...) Figure 5 This diagram illustrates a depth region capture method provided by an embodiment of the present invention. Figure 5 As shown, the area corresponding to the rectangle is the depth region in the depth map. Figure 5 The pedestrian in the image is bent over and facing to the right front, therefore the depth is selected. Figure 1 The lower left half of the area at a height of / 2, that is Figure 5 The area corresponding to the rectangle in the diagram.

[0099] After determining the depth region corresponding to the traveler, the first Gaussian curve in the horizontal direction and the second Gaussian curve in the vertical direction of each pixel in the depth region are further determined. The first Gaussian curve and the second Gaussian curve can be represented by the following formula (2):

[0100]

[0101] Where a represents the peak value of the Gaussian curve, b represents the peak position of the Gaussian curve, and c represents the half-peak width information of the Gaussian curve.

[0102] Optionally, step S23, determining the first Gaussian curve in the horizontal direction and the second Gaussian curve in the vertical direction for each pixel in the depth region, includes:

[0103] Sub-step S231: Based on the depth value of each pixel in the depth region, perform Gaussian fitting on each pixel in the depth region in the horizontal direction to obtain the first Gaussian curve of each pixel in the depth region in the horizontal direction.

[0104] Sub-step S232: Based on the depth value of each pixel in the depth region, perform Gaussian fitting on each pixel in the depth region in the vertical direction to obtain the second Gaussian curve of each pixel in the depth region in the vertical direction.

[0105] In practical applications, Gaussian fitting can be performed horizontally and vertically based on the depth values ​​of each pixel in the depth region, i.e., the distance between each pixel and the monocular camera, to obtain the first Gaussian curve and the second Gaussian curve. Specifically, the first Gaussian curve and the second Gaussian curve can be obtained by using the curve parameters a, b, and c in formula (2) to solve for the depth values ​​of each pixel in the depth region. The specific solution method can refer to the principle of Gaussian fitting, and will not be further elaborated in this embodiment of the invention.

[0106] After determining the first Gaussian curve and the second Gaussian curve, integrals are calculated over the first Gaussian curve and the second Gaussian curve respectively to obtain the first depth value and the second depth value. Specifically, the definite integral corresponding to the abscissa (x) of the first Gaussian curve is calculated to obtain the first depth value D1; the definite integral corresponding to the ordinate (f(x)) of the second Gaussian curve is calculated to obtain the second depth value D2.

[0107] By averaging the first depth value and the second depth value, we can obtain the first distance, which is the depth value of the pixel corresponding to the pedestrian on the depth map. The first distance reflects the distance between the pixel corresponding to the pedestrian and the monocular camera.

[0108] In this embodiment of the invention, when solving for the first distance, the Gaussian curve is integrated, which smooths out noise and outliers on the original depth map, improves the accuracy of the first distance, and helps to improve the accuracy of the final pedestrian distance measurement result.

[0109] In an optional embodiment of the present invention, step 105, calculating the second distance between the pedestrian and the monocular camera, includes:

[0110] Step S31: Determine the reference height of the pedestrian based on the pedestrian's posture information;

[0111] Step S32: Determine the pixel height value of the pedestrian in the vehicle's front view image based on the semantic information;

[0112] Step S33: Based on the pinhole imaging principle, calculate the second distance between the pedestrian and the monocular camera according to the focal length of the monocular camera, the reference height, and the pixel height value.

[0113] In this embodiment of the invention, different reference heights can be set for different postures. For example, referring to Table 3, one reference height setting strategy provided by this embodiment of the invention is shown.

[0114] Table 3

[0115] Standing 1.7m

[0116] bend over 1.3m Lying down 0.5m

[0117] It should be noted that the reference height can be set according to the actual test environment. Table 3 is only an exemplary illustration of the embodiment of the present invention and does not constitute a limitation on the specific value of the reference height in the present invention.

[0118] In this embodiment of the invention, the pixel height value of a pedestrian in the vehicle's front view image can be determined based on the semantic information of the vehicle's front view image. Specifically, pixels belonging to the category of "pedestrian" can be first determined based on the semantic information, and a pedestrian bounding box can be fitted based on these pixels belonging to the category of "pedestrian". The position coordinates of the pixels at the four vertices of the pedestrian bounding box can be used to determine the height of the pedestrian bounding box, that is, the pixel height value of the pedestrian in the vehicle's front view image. As an example, before calculating the second distance, this embodiment of the invention uses the semantic information of the vehicle's front view image to crop the target image of the pedestrian from the vehicle's front view image. The pixel height value of the pedestrian in the vehicle's front view image can be directly determined based on the position coordinates of the pixels in the cropped target image of the pedestrian. For example, if the pixels belonging to the category of "pedestrian" in the vehicle's front view image are concentrated in a rectangular area, and the coordinates of the four vertices of this rectangular area are (x1, y1), (x2, y1), (x1, y2), and (x2, y2), then the pixel height value of the pedestrian in the vehicle's front view image is h = y2 - y1.

[0119] Assuming the focal length of the monocular camera is F, the reference height of the pedestrian is H, and the pixel height of the pedestrian is h, then the second distance d2 can be expressed as:

[0120] d2=(F*H) / h (1)

[0121] After calculating the first distance and the second distance, the target distance between the pedestrian and the vehicle can be obtained by weighted summation of the first distance and the second distance.

[0122] In summary, the pedestrian ranging method provided by this invention performs image processing on the vehicle front view image using a first deep learning model to obtain semantic and depth information of the vehicle front view image. Then, based on the semantic information, a target image of the pedestrian is cropped from the vehicle front view image. The first distance is calculated using the depth information of the pixels in the cropped target image. This effectively reduces the influence of the background area in the vehicle front view image on the ranging result, and the depth information is more robust. Furthermore, the second distance between the pedestrian and the monocular camera is calibrated using the first distance calculated based on the depth information, which can effectively improve the accuracy of the pedestrian ranging result.

[0123] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0124] Reference Figure 6 The diagram illustrates a structural block diagram of an embodiment of a pedestrian ranging device according to the present invention. The device may include:

[0125] Image acquisition module 601 is used to acquire the front view image of the vehicle captured by a monocular camera;

[0126] The image processing module 602 is used to input the vehicle front view image into a pre-trained first deep learning model for semantic segmentation processing and image depth estimation processing to obtain the semantic information and depth information of the front view image;

[0127] Image cropping module 603 is used to crop a target image of a pedestrian from the front view image using the semantic information;

[0128] The first distance calculation module 604 is used to calculate the first distance between the pixels in the target image and the monocular camera based on the depth information.

[0129] The second distance calculation module 605 is used to calculate the second distance between the pedestrian and the monocular camera;

[0130] The target distance calculation module 606 is used to perform a weighted summation of the first distance and the second distance to obtain the target distance between the pedestrian and the vehicle.

[0131] Optionally, the first deep learning model includes a semantic segmentation branch and a depth estimation branch, wherein the semantic segmentation branch and the depth estimation branch reuse a feature extraction network and a feature aggregation network, the semantic segmentation branch further includes a category prediction network, and the depth estimation branch further includes a depth prediction network.

[0132] Optionally, the device further includes:

[0133] The training sample acquisition module is used to acquire a training sample set, which includes pedestrian sample images and semantic and depth labels of the pedestrian sample images;

[0134] The image output module is used to input the pedestrian sample image into the first deep learning model for processing in each round of training to obtain the semantic segmentation result and the first depth information output by the first deep learning model.

[0135] An information adjustment module is used to adjust the first depth information using the semantic segmentation result to obtain the second depth information;

[0136] The first loss value calculation module is used to calculate the first loss value of the semantic segmentation branch based on the semantic label and the semantic segmentation result, and to calculate the second loss value of the depth estimation branch based on the depth label and the second depth information;

[0137] The second loss value calculation module is used to perform a weighted summation of the first loss value and the second loss value to obtain the joint loss value of the first deep learning model.

[0138] The parameter adjustment module is used to adjust the model parameters of the first deep learning model according to the joint loss value, and to perform the next round of training until the joint loss value meets the preset conditions, so as to obtain the first deep learning model after training.

[0139] Optionally, the training sample acquisition module includes:

[0140] The data acquisition submodule is used to acquire pedestrian sample images using a monocular camera and to acquire three-dimensional data of pedestrians using radar; the monocular camera and the radar are coplanar.

[0141] The semantic label determination submodule is used to analyze the category to which each pixel in the pedestrian sample image belongs, and obtain the semantic label of the pedestrian sample image;

[0142] The depth label generation submodule is used to generate depth labels corresponding to the pedestrian sample images based on the 3D data.

[0143] The sample set construction submodule is used to construct a training sample set based on the collected pedestrian sample images and the semantic and depth labels of the pedestrian sample images.

[0144] Optionally, the depth information of the vehicle front view image includes a depth map corresponding to the vehicle front view image, and the first distance calculation module includes:

[0145] The prediction submodule is used to input the target image into the second deep learning model to perform pose prediction and orientation prediction, so as to obtain the pedestrian's pose information and orientation information.

[0146] The depth region determination submodule is used to determine the depth region corresponding to the pedestrian in the depth map based on the posture information and the orientation information.

[0147] The Gaussian curve determination submodule is used to determine the first Gaussian curve in the horizontal direction and the second Gaussian curve in the vertical direction for each pixel in the depth region.

[0148] The depth value calculation submodule is used to integrate the first Gaussian curve and the second Gaussian curve respectively to obtain the first depth value and the second depth value;

[0149] The first distance calculation submodule is used to calculate the average of the first depth value and the second depth value to obtain the first distance.

[0150] Optionally, the second distance calculation module includes:

[0151] The reference height determination submodule is used to determine the reference height of the pedestrian based on the pedestrian's posture information;

[0152] A pixel height value determination submodule is used to determine the pixel height value of the pedestrian in the vehicle front view image based on the semantic information.

[0153] The second distance calculation submodule is used to calculate the second distance between the pedestrian and the monocular camera based on the pinhole imaging principle, according to the focal length of the monocular camera, the reference height, and the pixel height value.

[0154] Optionally, the depth region determination submodule includes:

[0155] The first candidate region determination unit is used to determine the first candidate region to be cropped in the depth map according to the pose information and the preset first cropping strategy. The first cropping strategy is used to indicate the region to be cropped corresponding to each pose.

[0156] The second candidate region determination unit is used to determine the second candidate region to be cropped in the depth map according to the orientation information and the preset second cropping strategy. The second cropping strategy is used to indicate the region to be cropped corresponding to each orientation.

[0157] The depth region determination unit is used to find the intersection of the first candidate region and the second candidate region to obtain the depth region corresponding to the pedestrian in the depth map.

[0158] Optionally, the Gaussian curve determination submodule includes:

[0159] The first Gaussian fitting unit is used to perform Gaussian fitting on each pixel in the depth region in the horizontal direction based on the depth value of each pixel in the depth region, so as to obtain the first Gaussian curve of each pixel in the depth region in the horizontal direction.

[0160] The second Gaussian fitting unit is used to perform Gaussian fitting on each pixel in the depth region in the vertical direction based on the depth value of each pixel in the depth region, so as to obtain the second Gaussian curve of each pixel in the depth region in the vertical direction.

[0161] In summary, the pedestrian ranging device provided in this embodiment of the invention processes the vehicle front view image using a first deep learning model to obtain semantic and depth information of the vehicle front view image. Then, based on the semantic information, it crops the target image of the pedestrian from the vehicle front view image and calculates the first distance using the depth information of the pixels in the cropped target image. This effectively reduces the influence of the background area in the vehicle front view image on the ranging result, and the depth information is more robust. Furthermore, by using the first distance calculated based on the depth information to calibrate the second distance between the pedestrian and the monocular camera, the accuracy of the pedestrian ranging result can be effectively improved.

[0162] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0163] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0164] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0165] This invention also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by a processor of a device (server or terminal), the device is able to execute the foregoing text. Figure 1 The description of the pedestrian ranging method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0166] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0167] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0169] The present invention provides a detailed description of a pedestrian ranging method, apparatus, and readable storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for measuring pedestrian distance, characterized in that, The method includes: Acquire the front view image of the vehicle captured by a monocular camera; The vehicle front view image is input into a pre-trained first deep learning model for semantic segmentation and image depth estimation to obtain the semantic and depth information of the vehicle front view image. The semantic information is used to crop the target image of the pedestrian from the vehicle front view image; Calculate the first distance between the pixels in the target image and the monocular camera based on the depth information; Calculate the second distance between the pedestrian and the monocular camera; The first distance and the second distance are weighted and summed to obtain the target distance between the pedestrian and the vehicle. The depth information of the vehicle front view image includes a depth map corresponding to the vehicle front view image. The step of calculating the first distance between pixels in the target image and the monocular camera based on the depth information includes: The target image is input into a second deep learning model for pose and orientation prediction to obtain the pedestrian's pose and orientation information. Based on the pose information and a preset first cropping strategy, a first candidate region to be cropped in the depth map is determined, where the first cropping strategy indicates the region to be cropped for each pose. Based on the orientation information and a preset second cropping strategy, a second candidate region to be cropped in the depth map is determined, where the second cropping strategy indicates the region to be cropped for each orientation. The intersection of the first and second candidate regions is calculated to obtain the depth region corresponding to the pedestrian in the depth map. A first Gaussian curve in the horizontal direction and a second Gaussian curve in the vertical direction are determined for each pixel in the depth region. The first and second Gaussian curves are integrated to obtain a first depth value and a second depth value. The average of the first and second depth values ​​is calculated to obtain a first distance.

2. The method according to claim 1, characterized in that, The first deep learning model includes a semantic segmentation branch and a depth estimation branch. The semantic segmentation branch and the depth estimation branch reuse a feature extraction network and a feature aggregation network. The semantic segmentation branch also includes a category prediction network, and the depth estimation branch also includes a depth prediction network.

3. The method according to claim 2, characterized in that, Before inputting the vehicle front view image into a pre-trained first deep learning model for semantic segmentation and image depth estimation, the method further includes: Obtain a training sample set, which includes pedestrian sample images and semantic and depth labels of the pedestrian sample images; In each round of training, the pedestrian sample images are input into the first deep learning model for processing to obtain the semantic segmentation results and first depth information output by the first deep learning model. The first depth information is adjusted using the semantic segmentation result to obtain the second depth information; The first loss value of the semantic segmentation branch is calculated based on the semantic label and the semantic segmentation result, and the second loss value of the depth estimation branch is calculated based on the depth label and the second depth information. The first loss value and the second loss value are weighted and summed to obtain the joint loss value of the first deep learning model; The model parameters of the first deep learning model are adjusted according to the joint loss value, and the next round of training is carried out until the joint loss value meets the preset conditions, thus obtaining the first deep learning model after training.

4. The method according to claim 3, characterized in that, The acquisition of the training sample set includes: A monocular camera is used to capture sample images of pedestrians, and radar is used to capture three-dimensional data of the pedestrians; the monocular camera and the radar are coplanar. The semantic labels of the pedestrian sample images are obtained by analyzing the category to which each pixel belongs. Generate depth labels corresponding to the pedestrian sample images based on the three-dimensional data; A training sample set is constructed based on the collected pedestrian sample images and their semantic and depth labels.

5. The method according to claim 1, characterized in that, The calculation of the second distance between the pedestrian and the monocular camera includes: Determine the reference height of the pedestrian based on the pedestrian's posture information; The pixel height value of the pedestrian in the vehicle's front view image is determined based on the semantic information; Based on the pinhole imaging principle, the second distance between the pedestrian and the monocular camera is calculated according to the focal length of the monocular camera, the reference height, and the pixel height value.

6. The method according to claim 1, characterized in that, Determining the first Gaussian curve in the horizontal direction and the second Gaussian curve in the vertical direction for each pixel in the depth region includes: Based on the depth value of each pixel in the depth region, Gaussian fitting is performed on each pixel in the depth region in the horizontal direction to obtain the first Gaussian curve of each pixel in the depth region in the horizontal direction. Based on the depth value of each pixel in the depth region, Gaussian fitting is performed on each pixel in the depth region in the vertical direction to obtain the second Gaussian curve of each pixel in the depth region in the vertical direction.

7. A pedestrian distance measuring device, characterized in that, The device includes: The image acquisition module is used to acquire the front view image of the vehicle captured by the monocular camera; The image processing module is used to input the vehicle front view image into a pre-trained first deep learning model for semantic segmentation and image depth estimation to obtain the semantic information and depth information of the vehicle front view image. An image cropping module is used to crop a target image of a pedestrian from the vehicle front view image using the semantic information; The first distance calculation module is used to calculate the first distance between the pixels in the target image and the monocular camera based on the depth information. The second distance calculation module is used to calculate the second distance between the pedestrian and the monocular camera; The target distance calculation module is used to perform a weighted summation of the first distance and the second distance to obtain the target distance between the pedestrian and the vehicle; The depth information of the vehicle front view image includes a depth map corresponding to the vehicle front view image, and the first distance calculation module includes: The prediction submodule is used to input the target image into the second deep learning model to perform pose prediction and orientation prediction, so as to obtain the pedestrian's pose information and orientation information. The depth region determination submodule is used to determine the depth region corresponding to the pedestrian in the depth map based on the posture information and the orientation information. The Gaussian curve determination submodule is used to determine the first Gaussian curve in the horizontal direction and the second Gaussian curve in the vertical direction for each pixel in the depth region. The depth value calculation submodule is used to integrate the first Gaussian curve and the second Gaussian curve respectively to obtain the first depth value and the second depth value; The first distance calculation submodule is used to calculate the average of the first depth value and the second depth value to obtain the first distance; The depth region determination submodule includes: The first candidate region determination unit is used to determine the first candidate region to be cropped in the depth map according to the pose information and the preset first cropping strategy. The first cropping strategy is used to indicate the region to be cropped corresponding to each pose. The second candidate region determination unit is used to determine the second candidate region to be cropped in the depth map according to the orientation information and the preset second cropping strategy. The second cropping strategy is used to indicate the region to be cropped corresponding to each orientation. The depth region determination unit is used to find the intersection of the first candidate region and the second candidate region to obtain the depth region corresponding to the pedestrian in the depth map.

8. A machine-readable storage medium having instructions stored thereon that, when executed by one or more processors of a device, cause the device to perform the pedestrian ranging method as described in any one of claims 1 to 6.

Images