Image processing method and device, electronic equipment and vehicle

Abstract

The invention relates to an image processing method and device, electronic equipment and a vehicle, and is applied to the technical field of intelligent cabins. The method comprises the following steps: performing depth estimation on a cabin monocular image acquired by a monocular camera in a cabin based on a pre-trained depth estimation model to obtain a first depth image; determining a depth calibration parameter based on first depth information corresponding to the at least two reference points in the first depth image, an internal reference of the monocular camera and an actual distance between the at least two reference points; and calibrating depth information in the first depth image based on the depth calibration parameter to obtain a second depth image. According to the invention, the depth calibration parameter is determined by using the actual distance between the at least two reference points, and then the first depth image estimated according to the cabin monocular image is calibrated by using the depth calibration parameter, so that the more accurate second depth image is obtained, and the accuracy of the depth calibration can be improved under the condition that the hardware deployment cost is not increased. And the accuracy of depth estimation in a complex cabin scene is improved.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent cockpit technology, and more particularly to an image processing method, apparatus, electronic device, and vehicle. Background Technology

[0002] With the development of automotive intelligence, the level of intelligence in cockpits is also constantly improving. Among these technologies, 3D perception technology, as a crucial component of intelligent cockpits, can be applied to multiple areas such as human-vehicle interaction and safety monitoring. It optimizes the interactive experience, making user operation more convenient, while also ensuring the safety of passengers. Currently, 3D perception technology primarily relies on the acquisition of depth information. However, accurate depth information acquisition methods, such as binocular ranging, structured light ranging, and time-of-flight ranging, all require the deployment of complex hardware, resulting in high deployment costs. Meanwhile, single-view depth estimation methods have low accuracy in acquiring depth information in complex cockpit scenarios, making it difficult to meet practical needs. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides an image processing method, apparatus, electronic device, and vehicle.

[0004] A first aspect of this disclosure provides an image processing method, the method comprising: The depth of the cockpit monocular images captured by the monocular camera inside the cockpit is estimated based on the pre-trained depth estimation model to obtain the first depth image. Determine the depth calibration parameters based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between at least two reference points. The depth information in the first depth image is calibrated based on the depth calibration parameters to obtain the second depth image.

[0005] In some embodiments of this disclosure, depth calibration parameters are determined based on first depth information corresponding to at least two reference points in a first depth image, intrinsic parameters of a monocular camera, and the actual distance between the at least two reference points, including: Based on the first depth information corresponding to at least two reference points, determine the depth representation corresponding to at least two reference points, the depth representation including the parameter variables to be solved; Based on the depth representations corresponding to at least two reference points, the pixel coordinates of at least two reference points in the cockpit monocular image, and the intrinsic parameters of the monocular camera, determine the three-dimensional coordinate representations of at least two reference points. Determine the distance representation between at least two reference points based on the three-dimensional coordinate representation of at least two reference points; The depth calibration parameters are obtained by solving the parameter variables to be solved in the distance representation based on the actual distance between at least two reference points.

[0006] In some embodiments of this disclosure, the parameter variables to be solved in the distance representation are solved based on the actual distance between at least two reference points to obtain depth calibration parameters, including: The objective function is constructed with the goal of minimizing the absolute value of the difference between the actual distance and the distance representation. The parameter variables to be solved in the distance representation are then solved to obtain the depth calibration parameters.

[0007] In some embodiments of this disclosure, before obtaining a first depth image by performing depth estimation on cockpit monocular images captured by a monocular camera inside the cockpit based on a pre-trained depth estimation model, the method further includes: Construct a cockpit training sample set, which includes cockpit training images and corresponding calibration depth images. The cockpit training images are cockpit images acquired by a monocular camera. The depth estimation model is fine-tuned and trained based on the first sample in the cockpit training sample set. The fine-tuned depth estimation model is lightweighted based on the second sample in the cockpit training sample set. The lightweighting process includes at least one of model distillation and model quantization.

[0008] In some embodiments of this disclosure, after calibrating the depth information in the first depth image based on depth calibration parameters to obtain the second depth image, the method further includes: The cockpit monocular image is segmented based on a pre-trained image segmentation model to obtain segmentation results, which are used to indicate the position of each target object in the cockpit monocular image. Based on the segmentation results and the second depth image, extract the target depth information of each target object; Based on the target depth information and the intrinsic parameters of the monocular camera, point cloud data of each target object is generated; The distance between each target object is determined based on the point cloud data of each target object.

[0009] In some embodiments of this disclosure, after calibrating the depth information in the first depth image based on depth calibration parameters to obtain the second depth image, the method further includes: Hand keypoints are detected in the monocular images of the cockpit to obtain the pixel coordinates of the hand keypoints; The third depth information of the hand key points is determined based on the second depth image and the pixel coordinates of the hand key points; The 3D coordinates of the hand key points are determined based on the pixel coordinates of the hand key points, the third depth information, and the intrinsic parameters of the monocular camera. The gesture type is determined based on the three-dimensional coordinates corresponding to key points on the hand.

[0010] In some embodiments of this disclosure, the gesture type is determined based on the three-dimensional coordinates corresponding to key hand points, including: The finger curvature and finger orientation are determined based on the three-dimensional coordinates corresponding to key points of the hand; The gesture type is determined based on the degree of finger curvature and finger orientation.

[0011] A second aspect of this disclosure provides an image processing apparatus, the apparatus comprising: The estimation module is used to estimate the depth of the cockpit monocular images captured by the monocular camera inside the cockpit based on a pre-trained depth estimation model, and obtain the first depth image. The first determining module is used to determine depth calibration parameters based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between the at least two reference points. The calibration module is used to calibrate the depth information in the first depth image based on the depth calibration parameters to obtain the second depth image.

[0012] In some embodiments of this disclosure, the first determining module includes: The first determining unit is used to determine the depth representation corresponding to at least two reference points based on the first depth information corresponding to at least two reference points. The depth representation includes the parameter variables to be solved. The second determining unit is used to determine the three-dimensional coordinate representation of at least two reference points based on the depth representation corresponding to at least two reference points, the pixel coordinates of at least two reference points in the cockpit monocular image, and the intrinsic parameters of the monocular camera. The third determining unit is used to determine the distance representation between at least two reference points based on the three-dimensional coordinate representation of at least two reference points; The calculation unit is used to calculate the parameter variables to be calculated in the distance representation based on the actual distance between at least two reference points, so as to obtain the depth calibration parameters.

[0013] In some embodiments of this disclosure, the calculation unit is specifically used to construct an objective function with the goal of minimizing the absolute value of the difference between the actual distance and the distance representation, and to calculate the parameter variables to be calculated in the distance representation to obtain the depth calibration parameters.

[0014] In some embodiments of this disclosure, the image processing apparatus further includes a model adjustment module.

[0015] The model adjustment module is used to construct a cockpit training sample set, which includes cockpit training images and corresponding calibration depth images. The cockpit training images are cockpit images captured by a monocular camera. The depth estimation model is fine-tuned based on the first sample in the cockpit training sample set. The fine-tuned depth estimation model is then lightweighted based on the second sample in the cockpit training sample set. The lightweighting process includes at least one of model distillation and model quantization.

[0016] In some embodiments of this disclosure, the image processing apparatus further includes a second determining module.

[0017] The second determination module is used to segment the cockpit monocular image based on a pre-trained image segmentation model to obtain segmentation results, which are used to indicate the position of each target object in the cockpit monocular image; based on the segmentation results and the second depth image, the target depth information of each target object is extracted; based on the target depth information and the intrinsic parameters of the monocular camera, point cloud data of each target object is generated; and based on the point cloud data of each target object, the distance between each target object is determined.

[0018] In some embodiments of this disclosure, the image processing apparatus further includes a gesture recognition module.

[0019] The gesture recognition module is used to detect hand key points in the cockpit monocular image and obtain the pixel coordinates of the hand key points; determine the third depth information of the hand key points based on the second depth image and the pixel coordinates of the hand key points; determine the three-dimensional coordinates corresponding to the hand key points based on the pixel coordinates of the hand key points, the third depth information and the intrinsic parameters of the monocular camera; and determine the gesture type based on the three-dimensional coordinates corresponding to the hand key points.

[0020] In some embodiments of this disclosure, the gesture recognition module is specifically used to determine the finger curvature and finger orientation based on the three-dimensional coordinates corresponding to key points of the hand; and to determine the gesture type based on the finger curvature and finger orientation.

[0021] A third aspect of this disclosure provides an electronic device, including: a processor; and a memory for storing executable instructions; wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the image processing method provided in the first aspect.

[0022] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the image processing method provided in the first aspect.

[0023] A fifth aspect of this disclosure provides a computer program product comprising a computer program or instructions that, when executed by a processor, implement the image processing method of the first aspect described above.

[0024] A sixth aspect of this disclosure provides a vehicle that includes electronic equipment provided in the third aspect.

[0025] The technical solution provided in this disclosure has the following advantages compared with the prior art: In the image processing method, apparatus, electronic device, and vehicle provided in the embodiments of this disclosure, depth estimation is performed on the cockpit monocular image acquired by a monocular camera inside the cockpit based on a pre-trained depth estimation model to obtain a first depth image. Based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between the at least two reference points, depth calibration parameters are determined. Based on the depth calibration parameters, the depth information in the first depth image is calibrated to obtain a second depth image. The depth calibration parameters can be determined using the actual distance between at least two reference points, and then the first depth image estimated from the cockpit monocular image can be calibrated using the depth calibration parameters to obtain a more accurate second depth image. Thus, without deploying high-cost hardware such as binocular cameras or time-of-flight cameras, accurate and high-precision depth estimation results can be obtained using the cockpit monocular image acquired by the monocular camera. This improves the accuracy of single-view depth estimation in complex cockpit scenes and achieves low-cost, high-precision depth information acquisition. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this disclosure; Figure 2 This is a flowchart of a method for determining depth calibration parameters provided in an embodiment of this disclosure; Figure 3 This is a flowchart of a method for adjusting a depth estimation model provided in an embodiment of this disclosure; Figure 4This is a flowchart of a method for determining the distance between target objects provided in an embodiment of this disclosure; Figure 5 This is a flowchart of a gesture recognition method provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0030] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0031] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0032] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this disclosure. The method can be executed by an image processing device, which can be implemented in software and / or hardware, and can be configured in an in-vehicle infotainment system. Figure 1 As shown, the image processing method provided in this embodiment includes the following steps: S101. Based on the pre-trained depth estimation model, the depth of the cockpit monocular image acquired by the monocular camera inside the cockpit is estimated to obtain the first depth image.

[0033] The depth estimation model in this embodiment can be understood as an image processing model used to perform depth estimation processing on monocular images, determining the distance (i.e., depth information) from each object or surface in the scene corresponding to the monocular image to the monocular camera. The output of the depth estimation model is a depth image, where the pixel value of each pixel represents the distance from the corresponding position of that pixel in the original monocular image to the monocular camera. This distance is an absolute distance with a definite physical scale.

[0034] Optionally, the depth estimation model can be a large depth estimation model, such as the Metric3D series models, the Unidepth series models, and the Depthpro model, etc., without any limitation.

[0035] In this embodiment of the present disclosure, a monocular camera is fixedly installed inside the cockpit. The monocular camera can perform image acquisition operations periodically or when it receives an image acquisition command to obtain a monocular image of the cockpit and send the monocular image of the cockpit to the image processing device in the vehicle. After receiving the monocular image of the cockpit, the image processing device can use a pre-trained depth estimation model to perform depth estimation on the monocular image of the cockpit to obtain a first depth image.

[0036] In one exemplary embodiment of the present disclosure, when the image processing device performs depth estimation on the cockpit monocular image captured by the monocular camera inside the cockpit based on the pre-trained depth estimation model, it can input the cockpit monocular image into the depth estimation model and determine the output result of the depth estimation model as the first depth image.

[0037] Optionally, after acquiring the cockpit monocular image, the image processing device can determine whether depth estimation of the cockpit monocular image is needed based on the first acquisition time of the acquired cockpit monocular image. Specifically, it can determine whether the first acquisition time has reached the preset perception cycle based on the preset cockpit 3D perception strategy and the historical acquisition time of the cockpit monocular image that last performed depth estimation. If it has, the device will execute the step of performing depth estimation of the cockpit monocular image based on the pre-trained depth estimation model. If it has not, the device will discard the current cockpit monocular image and wait to acquire the next cockpit monocular image.

[0038] S102. Determine depth calibration parameters based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between at least two reference points.

[0039] In this embodiment, a reference point can be understood as an image point in a two-dimensional image that has significant, stable, and easily identifiable geometric or texture features, and whose position remains relatively stable in the three-dimensional space corresponding to the two-dimensional image. For example, a reference point can be a corner point of a car window, a corner point of a seat headrest, etc., and is not limited thereto.

[0040] The first depth information in this embodiment can be understood as the depth information corresponding to at least two reference points in the first depth image.

[0041] The intrinsic parameters in this embodiment may include focal length and principal point (optical center) coordinates.

[0042] The actual distance in this embodiment can be understood as the Euclidean distance between at least two positions corresponding to at least two reference points in three-dimensional space. This actual distance can be obtained in advance by measurement or calibration.

[0043] The depth calibration parameters in this embodiment can be understood as parameters used to calibrate the depth information determined based on the depth estimation model, so that the calibrated depth information is close to the true depth information. Optionally, the depth calibration parameters may include depth scaling factors, depth offsets, etc., which are not limited here.

[0044] In this embodiment, the image processing device can identify the pixel coordinates of at least two reference points from the cockpit monocular image based on the feature information of pre-acquired reference points, and extract the depth value corresponding to the pixel coordinates in the first depth image to obtain the first depth information of at least two reference points. The first depth information is calibrated using depth calibration parameters, and combined with the intrinsic parameters of the monocular camera, the pixel coordinates of the reference points in the image coordinate system are back-projected to the camera coordinate system to obtain the three-dimensional spatial coordinates of at least two reference points. The distance between the three-dimensional spatial coordinates of at least two reference points is determined as the theoretical distance. An iterative method is used to continuously adjust the depth calibration parameters so that the theoretical distance is as close as possible to the actual distance, and the depth calibration parameters that satisfy the iteration termination condition are determined as the final depth calibration parameters.

[0045] S103. Based on the depth calibration parameters, the depth information in the first depth image is calibrated to obtain the second depth image.

[0046] In this embodiment of the present disclosure, the image processing device can calibrate the depth information in the first depth image using the depth calibration parameters after determining the depth calibration parameters. Specifically, the depth information of each pixel in the first depth image can be calibrated, and the second depth image can be obtained by integrating the calibrated depth information.

[0047] This embodiment of the disclosure uses a pre-trained depth estimation model to estimate the depth of a cockpit monocular image captured by a monocular camera inside the cockpit, obtaining a first depth image. Based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between the at least two reference points, depth calibration parameters are determined. The depth information in the first depth image is calibrated based on the depth calibration parameters to obtain a second depth image. The actual distance between at least two reference points can be used to determine the depth calibration parameters, and then the first depth image estimated from the cockpit monocular image can be calibrated using the depth calibration parameters to obtain a more accurate second depth image. Thus, without deploying costly hardware such as binocular cameras or time-of-flight cameras, accurate and high-quality depth estimation results can be obtained using cockpit monocular images captured by a monocular camera, improving the accuracy of single-view depth estimation in complex cockpit scenarios.

[0048] Figure 2 This is a flowchart of a method for determining depth calibration parameters provided in an embodiment of this disclosure, such as... Figure 2 As shown, based on the above embodiments, the depth calibration parameters can be determined by the following method.

[0049] S201. Based on the first depth information corresponding to at least two reference points, determine the depth representation corresponding to at least two reference points, the depth representation including the parameter variables to be solved.

[0050] In this embodiment of the disclosure, there is a linear relationship between the first depth information and the depth representation of the reference point, for reference points among at least two reference points. The corresponding depth representation It can be represented as follows:

[0051] in, , These are the parameter variables to be solved. Reference point The corresponding first depth information.

[0052] The method of representing depth corresponding to other reference points among at least two reference points is the same as that of the reference points. The similarities are not elaborated here.

[0053] S202. Based on the depth representations corresponding to at least two reference points, the pixel coordinates of at least two reference points in the cockpit monocular image, and the intrinsic parameters of the monocular camera, determine the three-dimensional coordinate representations of at least two reference points.

[0054] In this embodiment of the disclosure, the reference point The pixel coordinates in the cockpit monocular image are Reference point Corresponding three-dimensional coordinate representation It can be represented as follows:

[0055] in, , The focal length is the intrinsic parameter of a monocular camera. , These are the coordinates of the principal point in the intrinsic parameters of a monocular camera.

[0056] The representation method of the three-dimensional coordinates of at least two reference points and other reference points. The similarities are not elaborated here.

[0057] S203. Based on the three-dimensional coordinate representation of at least two reference points, determine the distance representation between at least two reference points.

[0058] In this embodiment of the disclosure, at least two reference points may include reference points. and reference points Reference point The corresponding three-dimensional coordinate representation can be Reference point and reference points The distance between can be represented as .

[0059] S204. Solve the parameter variables to be solved in the distance representation based on the actual distance between at least two reference points to obtain the depth calibration parameters.

[0060] In this embodiment of the present disclosure, the image processing device can, after determining the distance representation between at least two reference points, calculate the parameter variables to be calculated in the distance representation based on the actual distance to obtain the depth calibration parameters.

[0061] In one exemplary embodiment of this disclosure, the image processing apparatus can input the actual distance and distance representation into a pre-trained parameter determination model, and the pre-trained parameter determination model can determine and output depth calibration parameters based on the input actual distance and distance representation.

[0062] In another exemplary embodiment of this disclosure, when there are three or more reference points, any two reference points can be combined into a group of reference points. Different groups of reference points may contain the same reference points, and each group of reference points corresponds to a set of actual distances and distance representations. Then, for each group of reference points, the image processing device can solve for a set of depth calibration parameters. After obtaining the depth calibration parameters corresponding to each group of reference points, multiple groups of depth calibration parameters can be fused, for example, by averaging, and the fusion result can be determined as the final depth calibration parameters.

[0063] This embodiment of the disclosure determines a depth representation corresponding to at least two reference points based on first depth information corresponding to at least two reference points. The depth representation includes parameter variables to be solved. Based on the depth representation corresponding to at least two reference points, the pixel coordinates of at least two reference points in the cockpit monocular image, and the intrinsic parameters of the monocular camera, a three-dimensional coordinate representation of at least two reference points is determined. Based on the three-dimensional coordinate representation of at least two reference points, a distance representation between at least two reference points is determined. Based on the actual distance between at least two reference points, the parameter variables to be solved in the distance representation are solved to obtain depth calibration parameters. This allows for accurate solution of depth calibration parameters using the known information of the actual distance between at least two reference points, reducing the estimation bias of the depth estimation model in single-view depth estimation, and further improving the accuracy of depth information acquisition.

[0064] In some embodiments, the depth calibration parameters are obtained by solving the parameter variables to be solved in the distance representation based on the actual distance between at least two reference points, including: constructing an objective function with the goal of minimizing the absolute value of the difference between the actual distance and the distance representation, solving the parameter variables to be solved in the distance representation, and obtaining the depth calibration parameters.

[0065] Specifically, the objective function can be expressed as follows:

[0066] in, For at least a reference point and reference points The actual distance between them.

[0067] Optionally, the image processing device can employ gradient descent, iteratively updating parameters along the negative gradient direction of the objective function to gradually approach the optimal solution. Alternatively, it can use particle swarm optimization, simulating the search process of a particle swarm in the solution space to find the global optimum. Other optimization algorithms can also be used, without limitation here.

[0068] Optionally, the image processing device may substitute the current solution into the distance representation during the solution process and calculate the absolute value of the difference between the actual distance and the distance representation. If the absolute value of the difference is less than a preset threshold, it is determined that the current solution meets the termination condition for optimization and the current solution is determined as the depth calibration parameter.

[0069] This embodiment constructs an objective function with the goal of minimizing the absolute value of the difference between the actual distance and the distance representation. It then calculates the parameter variables to be solved in the distance representation to obtain depth calibration parameters. This fully utilizes the actual distance as a benchmark value and minimizes the absolute value of the difference, so that the calculated depth calibration parameters closely match the real situation. This reduces the estimation bias of the depth estimation model in single-view depth estimation and further improves the accuracy of depth information acquisition.

[0070] Figure 3 This is a flowchart of a method for adjusting a depth estimation model provided in an embodiment of this disclosure, such as... Figure 3 As shown, based on the above embodiments, the depth estimation model can be adjusted using the following method.

[0071] S301. Construct a cockpit training sample set, which includes cockpit training images and corresponding calibration depth images. The cockpit training images are cockpit images acquired by a monocular camera.

[0072] The calibration depth image in this embodiment can be understood as the depth image corresponding to the cockpit training image used as calibration information.

[0073] In this embodiment of the present disclosure, the image processing device may first construct a cockpit training sample set for adjusting the depth estimation model, specifically by acquiring cockpit training images captured by a monocular camera and calibration depth images corresponding to each cockpit training image.

[0074] In one exemplary embodiment of this disclosure, the image processing device can acquire cockpit training images captured by a monocular camera and depth images captured by a time-of-flight camera. Based on the acquisition timestamps of the cockpit training images and depth images, the device performs time synchronization processing on the two to obtain multiple sets of aligned cockpit training images and depth images. Then, based on the intrinsic and extrinsic parameters of the monocular camera and the time-of-flight camera, the device registers the cockpit training images and depth images to obtain a calibration depth image corresponding to each cockpit training image.

[0075] S302. Fine-tune the depth estimation model based on the first sample in the cockpit training sample set.

[0076] In this embodiment of the present disclosure, after the cockpit training sample set is constructed, the image processing device may extract a portion of the samples as the first samples and fine-tune the depth estimation model based on the first samples.

[0077] In one exemplary embodiment of this disclosure, the image processing device can divide the first sample into a training set, a validation set, and a test set. The cockpit training images from the training set are input into the depth estimation model, and the loss between the predicted depth image output by the depth estimation model and the calibrated depth image is calculated. Specifically, a mean squared error loss function, a mean absolute value loss function, etc., can be used. Then, the backpropagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters. The model parameters are updated according to the gradient to minimize the loss value. During fine-tuning training, the performance of the depth estimation model is evaluated using the validation set to prevent overfitting. When the model's performance on the validation set reaches a satisfactory level or the number of training epochs reaches a preset value, fine-tuning training is stopped. Finally, the test set is used to perform a final evaluation of the fine-tuned model to ensure that the model has good generalization ability, thereby obtaining a depth estimation model suitable for cockpit scenarios.

[0078] S303. The fine-tuned depth estimation model is lightweighted based on the second sample in the cockpit training sample set. The lightweighting process includes at least one of model distillation and model quantization.

[0079] In this embodiment of the present disclosure, after the depth estimation model has been fine-tuned, the image processing device may extract a portion of samples from the cockpit training sample set as second samples, and perform lightweight processing on the depth estimation model based on the second samples.

[0080] In one exemplary embodiment of the present disclosure, the image processing device can determine whether to perform lightweight processing on the depth estimation model based on the computing power configuration of the vehicle's computer. When the computing power level of the vehicle's computer is lower than a preset level, the lightweight processing of the depth estimation model is performed.

[0081] Optionally, the image processing device can designate the fine-tuned depth estimation model as the teacher model and train a pre-defined student model during model distillation. Specifically, the cockpit training images from the second sample can be input into the teacher model and student model respectively. The teacher model outputs high-quality depth prediction results, while the student model outputs preliminary depth prediction results. The distillation loss between the student model's depth prediction results and the teacher model's depth prediction results, as well as the true loss between the student model's depth prediction results and the calibrated depth image, are calculated. The distillation loss and the true loss are weighted and summed according to preset weights to obtain the total loss function. The gradient of the total loss function with respect to the student model parameters is calculated using the backpropagation algorithm, and the model parameters are updated based on the gradient to minimize the loss value. During model distillation, the performance of the student model is estimated using a validation set to prevent overfitting. Model distillation stops when the model's performance on the validation set reaches a satisfactory level or the training epochs reach a preset value. Finally, the distilled student model is evaluated using a test set to ensure that the model has good generalization ability, and this distilled student model is designated as the updated depth estimation model.

[0082] Optionally, the image processing device can reduce the amount of data for the model parameters when quantizing the depth estimation model, for example, by converting it from floating-point numbers to low-precision (e.g., 8-bit integer) representations, continuously updating the quantized model parameters using second samples, allowing the model to learn and adapt in the quantized state, and using pre-set samples covering various common scenarios in the cockpit for quantization to minimize quantization loss.

[0083] This embodiment of the disclosure constructs a cockpit training sample set, which includes cockpit training images and corresponding calibration depth images. The cockpit training images are cockpit images captured by a monocular camera. Based on the first sample in the cockpit training sample set, the depth estimation model is fine-tuned and trained. Based on the second sample in the cockpit training sample set, the fine-tuned depth estimation model is lightweighted. The lightweighting process includes at least one of model distillation and model quantization. This enables the fine-tuned depth estimation model to highly fit the complex cockpit scene, accurately capture the details and spatial relationships of various objects in the cockpit, and further improve the accuracy of depth estimation. At the same time, the lightweighting process reduces the consumption of computing resources, speeds up inference, and makes the model easier to deploy on resource-constrained cockpit devices, achieving efficient and accurate depth estimation.

[0084] Figure 4 This is a flowchart illustrating a method for determining the distance between target objects according to an embodiment of this disclosure. Figure 4 As shown, based on the above embodiments, the distance between target objects can be determined by the following method.

[0085] S401. The cockpit monocular image is segmented based on the pre-trained image segmentation model to obtain the segmentation results, which are used to indicate the position of each target object in the cockpit monocular image.

[0086] The target object in this embodiment can be understood as an object inside the cockpit, such as the human body, seat, steering wheel, dashboard, etc., and is not limited thereto.

[0087] The image segmentation model in this embodiment can be understood as a deep learning algorithm model used to divide an image into multiple meaningful regions or objects. By performing fine classification of image pixels, it can accurately identify and distinguish object entities or texture regions with different semantics, assign a corresponding category label to each pixel, and finally generate a segmentation mask map with the same size as the input image. For example, the image segmentation model can be a Fully Convolutional Network (FCN) model, a U-Net model, etc., and is not limited thereto.

[0088] In this embodiment of the present disclosure, the image processing device can input the cockpit monocular image into a pre-trained image segmentation model, and the image segmentation model can perform recognition and segmentation processing on the cockpit monocular image to obtain a segmentation result that includes the position of each target object in the cockpit monocular image.

[0089] S402. Based on the segmentation results and the second depth image, extract the target depth information of each target object.

[0090] In this embodiment of the present disclosure, after obtaining the segmentation result output by the image segmentation model, the image processing device can extract the corresponding target depth information from the second depth image based on the pixel coordinates of each target object in the cockpit monocular image.

[0091] S403. Based on the target depth information and the intrinsic parameters of the monocular camera, generate point cloud data for each target object.

[0092] In this embodiment of the present disclosure, the image processing device can, for each target object, determine the three-dimensional coordinates of each pixel in the camera coordinate system by inverse transformation projection based on the pixel coordinates of each pixel corresponding to the target object in the cockpit monocular image and the target depth information of each pixel corresponding to the target object in the second depth image, combined with the intrinsic parameters of the monocular camera, and then construct point cloud data of the target object based on the three-dimensional coordinates.

[0093] S404. Determine the distance between each target object based on the point cloud data of each target object.

[0094] In this embodiment of the present disclosure, the image processing device can determine the distance between different target objects based on the point cloud data after the point cloud data of each target object has been constructed.

[0095] In one exemplary embodiment of this disclosure, the image processing device can determine the shortest distance between target objects based on the point cloud data of each target object. Specifically, for target object A and target object B, for each point of target object A, the nearest neighbor point is searched in the point cloud data of target object B, and the minimum distance is recorded. All recorded minimum distances are traversed, and the global minimum value is taken as the shortest distance between target object A and target object B. Target object A and target object B can be a human body and the passenger-side dashboard. After determining the shortest distance between the human body and the passenger-side dashboard, the seat position and angle can be adjusted according to a preset safety strategy. Alternatively, the airbag can be activated when the passenger's body is detected to be propelled forward due to sudden braking or a collision based on the shortest distance; this is not limited to this.

[0096] In another exemplary embodiment of this disclosure, the image processing device can determine the center distance between target objects based on the point cloud data of each target object. Specifically, for the point cloud data of target object A and target object B, the principal direction can be calculated using principal component analysis, and the center point of the target object can be determined by combining the point cloud density. The Euclidean distance between the corresponding center points of target object A and target object B can be calculated and determined as the center distance between target object A and target object B. Target object A and target object B can be the driver's hand and the steering wheel, respectively. According to a preset safety strategy, a safety warning can be issued when the center distance between the driver's hand and the steering wheel exceeds a preset safe distance; however, this is not limited to this.

[0097] This embodiment of the disclosure segments cockpit monocular images using a pre-trained image segmentation model to obtain segmentation results. The segmentation results indicate the position of each target object in the cockpit monocular image. Based on the segmentation results and a second depth image, target depth information of each target object is extracted. Based on the target depth information and the intrinsic parameters of the monocular camera, point cloud data of each target object is generated. The distance between each target object is determined based on the point cloud data of each target object. This allows for the reconstruction of the point cloud data of target objects within the cockpit based on depth information, thereby determining the accurate distance between each target object and providing a data foundation for high-precision spatial analysis and interactive tasks.

[0098] Figure 5 This is a flowchart of a gesture recognition method provided in an embodiment of this disclosure, such as... Figure 5 As shown, based on the above embodiments, gesture recognition can be performed using the following method.

[0099] S501. Perform hand key point detection on the monocular image of the cockpit to obtain the pixel coordinates of the hand key points.

[0100] The key points of the hand in this embodiment can be understood as a set of semantically meaningful feature points used to describe the posture and shape of the hand, which can accurately mark the position of various parts of the hand (such as finger joints, palm outline, etc.).

[0101] In this embodiment of the present disclosure, the image processing device can perform hand key point detection on the cockpit monocular image. Specifically, a pre-trained key point detection model can be used. The cockpit monocular image is input into the key point detection model, and the key point detection model identifies the hand key points from the monocular cockpit image based on the feature information of the pre-learned hand key points, and outputs the pixel coordinates of the hand key points.

[0102] S502. Determine the third depth information of the hand key points based on the second depth image and the pixel coordinates of the hand key points.

[0103] In this embodiment of the present disclosure, the image processing device can extract the corresponding third depth information from the second depth image based on the pixel coordinates of the key points of the hand after obtaining the pixel coordinates.

[0104] S503. Determine the 3D coordinates of the hand key points based on the pixel coordinates of the hand key points, the third depth information, and the intrinsic parameters of the monocular camera.

[0105] In this embodiment of the present disclosure, the image processing device can determine the three-dimensional coordinates of the hand key points in the camera coordinate system by inverse transformation projection based on the pixel coordinates of the hand key points, the third depth information, and the intrinsic parameters of the monocular camera.

[0106] S504. Determine the gesture type based on the three-dimensional coordinates corresponding to the key points of the hand.

[0107] The gesture type in this disclosure can be understood as a classification of specific semantics or intentions conveyed through hand movements and postures. For example, the gesture type may include fist gesture type, outstretched gesture type, pointing gesture type, number gesture type, symbol gesture type, etc., without limitation.

[0108] In this embodiment of the present disclosure, the image processing device can determine the user's gesture type based on the three-dimensional coordinates corresponding to the key points of the hand after determining the three-dimensional coordinates corresponding to the key points of the hand.

[0109] In one exemplary embodiment of this disclosure, the image processing device can extract features from the three-dimensional coordinates corresponding to key points on the hand, and match the extracted features with features of preset gesture types to determine the gesture type of the three-dimensional coordinates corresponding to the key points on the hand.

[0110] This embodiment of the present disclosure detects hand key points in a cockpit monocular image to obtain the pixel coordinates of the hand key points. Based on the second depth image and the pixel coordinates of the hand key points, the third depth information of the hand key points is determined. Based on the pixel coordinates of the hand key points, the third depth information, and the intrinsic parameters of the monocular camera, the three-dimensional coordinates corresponding to the hand key points are determined. Based on the three-dimensional coordinates corresponding to the hand key points, the gesture type is determined. This method can reconstruct the three-dimensional coordinates of the hand key points based on the depth information, and then determine the gesture type based on the richer information in the three-dimensional coordinates of the hand key points, thereby improving the accuracy of gesture recognition and providing a data foundation for high-precision human-vehicle interaction tasks.

[0111] In some embodiments, the image processing device may reconstruct the point cloud data of each target object based on the target depth information and the intrinsic parameters of the monocular camera, and then determine the gesture type based on the point cloud data corresponding to the hand object. Alternatively, it may filter the point cloud data of key hand points from the point cloud data corresponding to the hand object, and then determine the gesture type based on the point cloud data of the key hand points. No limitation is made here.

[0112] In some embodiments, determining the gesture type based on the three-dimensional coordinates corresponding to key points of the hand includes: determining the finger curvature and finger orientation based on the three-dimensional coordinates corresponding to key points of the hand; and determining the gesture type based on the finger curvature and finger orientation.

[0113] Specifically, the image processing device can use a three-dimensional vector angle formula to calculate the angle between the lines connecting adjacent key points, thereby determining the finger curvature. It then determines the finger's pointing direction based on the direction of the line connecting the fingertip key point and the finger root key point, and calculates the angle between the finger's pointing direction and the spatial coordinate axes of the camera coordinate system to obtain the finger orientation. After determining the finger curvature and finger orientation, it further matches them with preset finger curvature and finger orientation for various gesture types to determine the gesture type corresponding to the three-dimensional coordinates of the hand key points.

[0114] This embodiment of the invention determines the finger curvature and finger orientation based on the three-dimensional coordinates corresponding to key points on the hand, and determines the gesture type based on the finger curvature and finger orientation. This enables more refined gesture recognition based on the finger curvature and finger orientation, further improving the accuracy of gesture recognition.

[0115] Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure. Figure 6As shown, the image processing device 600 includes: an estimation module 610, a first determination module 620, and a calibration module 630. The estimation module 610 is used to estimate the depth of a monocular image captured by a monocular camera inside the cockpit based on a pre-trained depth estimation model, to obtain a first depth image. The first determination module 620 is used to determine depth calibration parameters based on first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between at least two reference points. The calibration module 630 is used to calibrate the depth information in the first depth image based on the depth calibration parameters, to obtain a second depth image.

[0116] This embodiment of the disclosure uses a pre-trained depth estimation model to estimate the depth of a cockpit monocular image captured by a monocular camera inside the cockpit, obtaining a first depth image. Based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between the at least two reference points, depth calibration parameters are determined. The depth information in the first depth image is calibrated based on the depth calibration parameters to obtain a second depth image. The actual distance between at least two reference points can be used to determine the depth calibration parameters, and then the first depth image estimated from the cockpit monocular image can be calibrated using the depth calibration parameters to obtain a more accurate second depth image. Thus, without deploying costly hardware such as binocular cameras or time-of-flight cameras, accurate and high-quality depth estimation results can be obtained using cockpit monocular images captured by a monocular camera, improving the accuracy of single-view depth estimation in complex cockpit scenarios.

[0117] Optionally, the first determining module 620 includes: a first determining unit, configured to determine a depth representation corresponding to at least two reference points based on first depth information corresponding to at least two reference points, the depth representation including parameter variables to be solved; a second determining unit, configured to determine a three-dimensional coordinate representation of at least two reference points based on the depth representation corresponding to at least two reference points, the pixel coordinates of at least two reference points in the cockpit monocular image, and the intrinsic parameters of the monocular camera; a third determining unit, configured to determine a distance representation between at least two reference points based on the three-dimensional coordinate representation of at least two reference points; and a solving unit, configured to solve the parameter variables to be solved in the distance representation based on the actual distance between at least two reference points to obtain depth calibration parameters.

[0118] Optionally, the solution unit is specifically used to construct an objective function with the goal of minimizing the absolute value of the difference between the actual distance and the distance representation, and to solve the parameter variables to be solved in the distance representation to obtain the depth calibration parameters.

[0119] Optionally, the image processing device 600 further includes a model adjustment module. The model adjustment module is used to construct a cockpit training sample set, which includes cockpit training images and corresponding calibration depth images, wherein the cockpit training images are cockpit images acquired by a monocular camera; to fine-tune the depth estimation model based on a first sample in the cockpit training sample set; and to perform lightweight processing on the fine-tuned depth estimation model based on a second sample in the cockpit training sample set, wherein the lightweight processing includes at least one of model distillation and model quantization.

[0120] Optionally, the image processing device 600 further includes a second determining module. The second determining module is used to segment the cockpit monocular image based on a pre-trained image segmentation model to obtain segmentation results, which are used to indicate the position of each target object in the cockpit monocular image; based on the segmentation results and a second depth image, to extract target depth information for each target object; based on the target depth information and the intrinsic parameters of the monocular camera, to generate point cloud data for each target object; and based on the point cloud data of each target object, to determine the distance between each target object.

[0121] Optionally, the image processing device 600 also includes a gesture recognition module. The gesture recognition module is used to detect hand key points in the cockpit monocular image to obtain the pixel coordinates of the hand key points; determine third depth information of the hand key points based on the second depth image and the pixel coordinates of the hand key points; determine the three-dimensional coordinates corresponding to the hand key points based on the pixel coordinates of the hand key points, the third depth information, and the intrinsic parameters of the monocular camera; and determine the gesture type based on the three-dimensional coordinates corresponding to the hand key points.

[0122] Optionally, the gesture recognition module is specifically used to determine the finger curvature and finger orientation based on the three-dimensional coordinates corresponding to key points on the hand; and to determine the gesture type based on the finger curvature and finger orientation.

[0123] The image processing apparatus provided in this embodiment can execute the methods of any of the above embodiments, and its execution method and beneficial effects are similar, so they will not be described again here.

[0124] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0125] In this embodiment of the disclosure, Figure 7 The electronic devices shown can be servers or terminals. Specifically, terminals include in-vehicle systems, computers, or tablets, etc., without limitation.

[0126] like Figure 7 As shown, the electronic device may include a processor 710 and a memory 720 storing computer program instructions.

[0127] Specifically, the processor 710 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this disclosure.

[0128] Memory 720 may include a large-capacity storage for information or instructions. For example, and not limitingly, memory 720 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 720 may include removable or non-removable (or fixed) media. Where appropriate, memory 720 may be internal or external to the integrated gateway device. In a particular embodiment, memory 720 is a non-volatile solid-state memory. In a particular embodiment, memory 720 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0129] The processor 710 reads and executes computer program instructions stored in the memory 720 to perform the steps of the image processing method provided in the embodiments of this disclosure.

[0130] In one example, the electronic device may also include a transceiver 730 and a bus 740. Wherein, as... Figure 7 As shown, the processor 710, memory 720 and transceiver 730 are connected via bus 740 and communicate with each other.

[0131] Bus 740 may include hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 740 may include one or more buses.

[0132] This disclosure also provides a computer-readable storage medium that can store a computer program, which, when executed by a processor, causes the processor to implement the image processing method provided in this disclosure.

[0133] The computer program, when executed by the processor, performs the following steps: performing depth estimation on the monocular images of the cockpit captured by the monocular camera inside the cockpit based on a pre-trained depth estimation model to obtain a first depth image; determining depth calibration parameters based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between at least two reference points; and calibrating the depth information in the first depth image based on the depth calibration parameters to obtain a second depth image.

[0134] It can determine the depth calibration parameters using the actual distance between at least two reference points, and then use the depth calibration parameters to calibrate the first depth image estimated from the cockpit monocular image to obtain a more accurate second depth image. Thus, without deploying high-cost hardware such as binocular cameras and time-of-flight cameras, it can obtain accurate and high-quality depth estimation results using cockpit monocular images acquired by a monocular camera, thereby improving the accuracy of single-view depth estimation in complex cockpit scenarios.

[0135] In some embodiments of this disclosure, determining depth calibration parameters based on first depth information corresponding to at least two reference points in a first depth image, intrinsic parameters of a monocular camera, and the actual distance between the at least two reference points includes: determining a depth representation corresponding to at least two reference points based on the first depth information corresponding to the at least two reference points, the depth representation including parameter variables to be solved; determining a three-dimensional coordinate representation of at least two reference points based on the depth representation corresponding to the at least two reference points, the pixel coordinates of the at least two reference points in the cockpit monocular image, and the intrinsic parameters of the monocular camera; determining a distance representation between the at least two reference points based on the three-dimensional coordinate representation; and solving the parameter variables to be solved in the distance representation based on the actual distance between the at least two reference points to obtain the depth calibration parameters.

[0136] In some embodiments of this disclosure, the depth calibration parameters are obtained by solving the parameter variables to be solved in the distance representation based on the actual distance between at least two reference points. This includes: constructing an objective function with the goal of minimizing the absolute value of the difference between the actual distance and the distance representation, solving the parameter variables to be solved in the distance representation, and obtaining the depth calibration parameters.

[0137] In some embodiments of this disclosure, before obtaining a first depth image by performing depth estimation on a cockpit monocular image captured by a monocular camera inside the cockpit based on a pre-trained depth estimation model, the method further includes: constructing a cockpit training sample set, which includes cockpit training images and corresponding calibration depth images, wherein the cockpit training images are cockpit images captured by a monocular camera; fine-tuning the depth estimation model based on the first sample in the cockpit training sample set; and performing lightweight processing on the fine-tuned depth estimation model based on the second sample in the cockpit training sample set, wherein the lightweight processing includes at least one of model distillation and model quantization.

[0138] In some embodiments of this disclosure, after calibrating the depth information in the first depth image based on depth calibration parameters to obtain the second depth image, the method further includes: segmenting the cockpit monocular image based on a pre-trained image segmentation model to obtain a segmentation result, the segmentation result being used to indicate the position of each target object in the cockpit monocular image; extracting target depth information of each target object based on the segmentation result and the second depth image; generating point cloud data of each target object based on the target depth information and the intrinsic parameters of the monocular camera; and determining the distance between each target object based on the point cloud data of each target object.

[0139] In some embodiments of this disclosure, after calibrating the depth information in the first depth image based on depth calibration parameters to obtain the second depth image, the method further includes: detecting hand key points in the cockpit monocular image to obtain the pixel coordinates of the hand key points; determining third depth information of the hand key points based on the second depth image and the pixel coordinates of the hand key points; determining the three-dimensional coordinates corresponding to the hand key points based on the pixel coordinates of the hand key points, the third depth information, and the intrinsic parameters of the monocular camera; and determining the gesture type based on the three-dimensional coordinates corresponding to the hand key points.

[0140] In some embodiments of this disclosure, the gesture type is determined based on the three-dimensional coordinates corresponding to key points of the hand, including: determining the finger curvature and finger orientation based on the three-dimensional coordinates corresponding to key points of the hand; and determining the gesture type based on the finger curvature and finger orientation.

[0141] The aforementioned storage medium may, for example, include a memory 720 for computer program instructions, which can be executed by the processor 710 of the image processing device to complete the image processing method provided in the embodiments of this disclosure. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc ROM (CD-ROM), magnetic tape, floppy disk, and optical data storage device. The aforementioned computer program may be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0142] This disclosure also provides a vehicle that includes electronic devices that can implement the various processes and effects described in the above embodiments of this disclosure, which will not be elaborated here.

[0143] This disclosure also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, they implement the image processing method provided in this disclosure and can achieve the various processes and effects in the above embodiments of this disclosure, which will not be elaborated here.

[0144] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0145] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, The method includes: The depth of the cockpit monocular images captured by the monocular camera inside the cockpit is estimated based on the pre-trained depth estimation model to obtain the first depth image. Depth calibration parameters are determined based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between the at least two reference points. The depth information in the first depth image is calibrated based on the depth calibration parameters to obtain the second depth image.

2. The method according to claim 1, characterized in that, The step of determining depth calibration parameters based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between the at least two reference points includes: Based on the first depth information corresponding to the at least two reference points, a depth representation corresponding to the at least two reference points is determined, wherein the depth representation includes parameter variables to be solved; Based on the depth representation corresponding to the at least two reference points, the pixel coordinates of the at least two reference points in the cockpit monocular image, and the intrinsic parameters of the monocular camera, the three-dimensional coordinate representation of the at least two reference points is determined; Based on the three-dimensional coordinate representation of the at least two reference points, determine the distance representation between the at least two reference points; The depth calibration parameters are obtained by solving the parameter variables to be solved in the distance representation based on the actual distance between the at least two reference points.

3. The method according to claim 2, characterized in that, The process of solving the parameter variables to be solved in the distance representation based on the actual distance between the at least two reference points to obtain the depth calibration parameters includes: The objective function is constructed with the goal of minimizing the absolute value of the difference between the actual distance and the distance representation. The parameter variables to be solved in the distance representation are then solved to obtain the depth calibration parameters.

4. The method according to claim 1, characterized in that, Before obtaining the first depth image by performing depth estimation on the cockpit monocular images captured by the pre-trained depth estimation model based on the monocular camera inside the cockpit, the method further includes: Construct a cockpit training sample set, which includes cockpit training images and calibration depth images corresponding to the cockpit training images, wherein the cockpit training images are cockpit images acquired by the monocular camera; The depth estimation model is fine-tuned and trained based on the first sample in the cockpit training sample set. The fine-tuned depth estimation model is lightweighted based on the second sample in the cockpit training sample set. The lightweighting process includes at least one of model distillation and model quantization.

5. The method according to claim 1, characterized in that, After calibrating the depth information in the first depth image based on the depth calibration parameters to obtain the second depth image, the method further includes: The cockpit monocular image is segmented based on a pre-trained image segmentation model to obtain segmentation results, which are used to indicate the position of each target object in the cockpit monocular image. Based on the segmentation results and the second depth image, the target depth information of each target object is extracted; Based on the target depth information and the intrinsic parameters of the monocular camera, point cloud data of each target object is generated; The distance between the target objects is determined based on the point cloud data of each target object.

6. The method according to claim 1, characterized in that, After calibrating the depth information in the first depth image based on the depth calibration parameters to obtain the second depth image, the method further includes: Hand key points are detected in the monocular image of the cockpit to obtain the pixel coordinates of the hand key points; The third depth information of the hand key points is determined based on the second depth image and the pixel coordinates of the hand key points; The three-dimensional coordinates corresponding to the hand key points are determined based on the pixel coordinates of the hand key points, the third depth information, and the intrinsic parameters of the monocular camera. The gesture type is determined based on the three-dimensional coordinates corresponding to the key points of the hand.

7. The method according to claim 6, characterized in that, The determination of gesture type based on the three-dimensional coordinates corresponding to the key hand points includes: The finger curvature and finger orientation are determined based on the three-dimensional coordinates corresponding to the key points of the hand. The gesture type is determined based on the degree of finger curvature and finger orientation.

8. An image processing apparatus, characterized in that, The device includes: The estimation module is used to estimate the depth of the cockpit monocular images captured by the monocular camera inside the cockpit based on a pre-trained depth estimation model, and obtain the first depth image. The first determining module is used to determine depth calibration parameters based on the first depth information corresponding to at least two reference points in the first depth image, the intrinsic parameters of the monocular camera, and the actual distance between the at least two reference points. The calibration module is used to calibrate the depth information in the first depth image based on the depth calibration parameters to obtain a second depth image.

9. An electronic device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the image processing method according to any one of claims 1-7.

10. A vehicle, characterized in that, Including the electronic device as described in claim 9.

Images