Image processing method, device, electronic device and storage medium

Abstract

The present application relates to image analysis technology and provides an image processing method, device, electronic device, and storage medium. The image processing method includes: obtaining a driving left image and a driving right image of a driving vehicle during driving, generating an instance segmented left image corresponding to the driving left image and an instance segmented right image corresponding to the driving right image, reconstructing the driving left image based on an autoencoder to obtain a predicted disparity map, generating a first error value based on the driving left image, the predicted disparity map, and the driving right image, and generating a second error value based on the instance segmented left image, the predicted disparity map, and the instance segmented right image, adjusting the autoencoder based on the first error value and the second error value to obtain an autoencoder model, obtaining a test driving image of the vehicle to be tested during driving, reconstructing the test driving image based on the autoencoder model to obtain a target disparity map, and converting the target disparity map to generate a refined depth image.

Description

Technical Field

[0001] The present application relates to the field of image analysis technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] When a vehicle is in autonomous driving, it needs to use depth images to constantly monitor pedestrians, vehicles, or obstacles around it. However, the accuracy and fineness of depth images generated requires a large amount of data calculation, which affects autonomous driving. Summary of the Invention

[0003] In view of the above, it is necessary to provide an image processing method, device, electronic device and storage medium that can generate accurate depth images to avoid affecting the automatic driving of the vehicle.

[0004] The present application provides an image processing method, which includes:

[0005] Acquire a vehicle driving image of the vehicle during driving, wherein the vehicle driving image includes a left driving image and a right driving image;

[0006] performing instance segmentation processing on the vehicle driving image to obtain an instance segmented image, wherein the instance segmented image includes an instance segmented left image corresponding to the driving left image and an instance segmented right image corresponding to the driving right image;

[0007] Reconstructing the left driving image based on a pre-built autoencoder to obtain a predicted disparity map;

[0008] generating a first error value of the autoencoder for the vehicle driving image according to the driving left image, the predicted disparity map, and the driving right image, and generating a second error value of the autoencoder for the instance segmented image according to the instance segmented left image, the predicted disparity map, and the instance segmented right image;

[0009] Adjust the autoencoder according to the first error value and the second error value to obtain an autoencoder model;

[0010] Acquire a test driving image of the vehicle to be tested during driving, and reconstruct the test driving image according to the autoencoder model to obtain a target disparity map;

[0011] The target disparity map is converted to obtain a depth image corresponding to the test driving image.

[0012] According to an optional embodiment of the present application, obtaining a vehicle driving image of a moving vehicle during driving includes one or a combination of the following methods:

[0013] Controlling a binocular camera to capture the field of view of the moving vehicle during the moving process to obtain the moving left image and the moving right image;

[0014] An image corresponding to the moving vehicle is obtained from a first preset database as the moving left image, a shooting time of the moving left image is obtained from the first preset database, and an image corresponding to both the moving vehicle and the shooting time is obtained from a second preset database as the moving right image.

[0015] According to an optional embodiment of the present application, performing instance segmentation processing on the vehicle driving image to obtain an instance segmented image includes:

[0016] Inputting the vehicle driving image into a pre-trained instance segmentation model to obtain an output image;

[0017] Detecting the segmentation accuracy of the instance segmentation model on the vehicle driving image based on the output image;

[0018] If the segmentation accuracy is greater than a preset threshold, the output image is determined as the instance segmented image.

[0019] According to an optional embodiment of the present application, the autoencoder includes an encoder and a decoder, and the reconstructing the driving left image based on the pre-built autoencoder to obtain the predicted disparity map includes:

[0020] performing encoding processing on the traveling left image based on the encoder to obtain a feature vector of the traveling left image;

[0021] Mapping processing is performed on the feature vector based on the decoder to obtain the predicted disparity map.

[0022] According to an optional embodiment of the present application, generating a first error value of the autoencoder for the vehicle driving image according to the driving left image, the predicted disparity map, and the driving right image includes:

[0023] Obtaining a predicted disparity value corresponding to each pixel in the driving left image from the predicted disparity map;

[0024] Locating a first pixel position of each pixel in the left driving image;

[0025] determining a second pixel position based on the predicted disparity value and the first pixel position;

[0026] Obtaining an RGB pixel value of a pixel point corresponding to the first pixel position from the left driving image;

[0027] Generate a predicted right image of the driving left image by the autoencoder according to the RGB pixel value of the pixel point corresponding to the first pixel position and the second pixel position;

[0028] Generating the first error value according to the driving right image and the predicted right image includes:

[0029]

[0030] Wherein, MSE refers to the first error value, y is the RGB pixel value of the right driving image, is the RGB pixel value of the predicted right image.

[0031] According to an optional embodiment of the present application, adjusting the autoencoder according to the first error value and the second error value to obtain the autoencoder model includes:

[0032] Calculating the sum of the first error value and the second error value to obtain a total training error value of the autoencoder;

[0033] The configuration parameters in the autoencoder are adjusted until the total training error value no longer decreases, thereby obtaining the autoencoder model.

[0034] According to an optional embodiment of the present application, converting the target disparity map to obtain a depth image corresponding to the test driving image includes:

[0035] Obtaining a focal length value of a shooting device for the test driving image, and obtaining a lens distance between lenses in the shooting device;

[0036] Acquire a target disparity value of each pixel in the test driving image from the target disparity map;

[0037] The depth pixel value corresponding to each pixel point in the test driving image is calculated according to the focal length value, the lens distance and the target parallax value. The calculation formula of the depth pixel value is:

[0038]

[0039] Wherein, z refers to the depth pixel value, b refers to the lens distance, f refers to the focal length value, and d refers to the target disparity value;

[0040] The depth image is generated according to each pixel point in the test driving image and the depth pixel value.

[0041] The present application provides an image processing device, comprising:

[0042] an acquisition unit, configured to acquire a vehicle driving image of the vehicle during driving, wherein the vehicle driving image includes a left driving image and a right driving image;

[0043] a segmentation unit, configured to perform instance segmentation processing on the vehicle driving image to obtain instance segmented images, wherein the instance segmented images include an instance segmented left image corresponding to the vehicle driving left image and an instance segmented right image corresponding to the vehicle driving right image;

[0044] A reconstruction unit, configured to reconstruct the left driving image based on a pre-built autoencoder to obtain a predicted disparity map;

[0045] a generating unit, configured to generate a first error value of the autoencoder for the vehicle driving image based on the driving left image, the predicted disparity map, and the driving right image, and to generate a second error value of the autoencoder for the instance segmented image based on the instance segmented left image, the predicted disparity map, and the instance segmented right image;

[0046] an adjusting unit, configured to adjust the autoencoder according to the first error value and the second error value to obtain an autoencoder model;

[0047] The reconstruction unit is further configured to obtain a test driving image of the vehicle under test during driving, and reconstruct the test driving image according to the autoencoder model to obtain a target disparity map;

[0048] A conversion unit is used to convert the target disparity map to obtain a depth image corresponding to the test driving image.

[0049] The present application provides an electronic device, comprising:

[0050] a memory storing at least one instruction; and

[0051] A processor executes the at least one instruction to implement the image processing method.

[0052] The present application provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the image processing method.

[0053] It can be seen from the above technical solution that the present application can accurately identify the label information in the vehicle driving image by performing instance segmentation processing on the vehicle driving image, and analyze the first error value of the autoencoder for the vehicle driving image and the second error value of the autoencoder for the instance segmented image through the predicted disparity map generated by encoding the driving left image, thereby enhancing the consistency of the predicted disparity map on the same object, and then eliminating the interference of the autoencoder on image encoding, and improving the accuracy of the first error value and the second error value. By adjusting the autoencoder in combination with the first error value and the second error value, the image processing capability of the autoencoder model can be further improved, so that the test driving image can be processed based on the autoencoder model, and the accuracy and fineness of the depth image can be improved, thereby improving the accuracy of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is an application environment diagram of a preferred embodiment of the image processing method of the present application.

[0055] Figure 2 It is a flow chart of a preferred embodiment of the image processing method of the present application.

[0056] Figure 3 This is a functional module diagram of a preferred embodiment of the image processing device of the present application.

[0057] Figure 4 It is a structural diagram of an electronic device for implementing a preferred embodiment of the image processing method of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of this application clearer, this application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] like Figure 1 , which is an application environment diagram of a preferred embodiment of an image processing method of the present application. The image processing method can be applied to one or more electronic devices 1, wherein the electronic device 1 communicates with a binocular camera 2 and a photographing device 3. The binocular camera 2 and the photographing device 3 can be any device capable of photographing.

[0060] The image processing method can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0061] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0062] The image processing method is applied to one or more electronic devices, which are devices that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions. Their hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0063] The electronic device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0064] The electronic device may include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network electronic device, a group of electronic devices consisting of multiple network electronic devices, or a cloud based on cloud computing consisting of a large number of hosts or network electronic devices.

[0065] The network where the electronic device is located includes, but is not limited to: the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0066] like Figure 2 The figure is a flow chart of a preferred embodiment of an image processing method of the present application. According to different requirements, the order of each step in the flow chart can be adjusted according to the actual detection requirements, and some steps can be omitted. The execution subject of the method is an electronic device, such as Figure 1 The electronic device 1 is shown.

[0067] S10 , obtaining a vehicle driving image of the vehicle during the driving process, wherein the vehicle driving image includes a left driving image and a right driving image.

[0068] In at least one embodiment of the present application, the traveling vehicle may refer to any vehicle that requires automatic driving.

[0069] The vehicle driving image refers to an image generated by a binocular camera during the driving process of the vehicle, wherein the binocular camera includes two lenses.

[0070] The left driving image refers to an image captured by the left lens of the binocular camera of the field of view of the moving vehicle during travel. The left driving image can be obtained from a first preset database associated with the left lens. The first preset database stores multiple images captured by the left lens and the capture time of each image.

[0071] The driving right image refers to an image captured by the right lens of the binocular camera of the field of view of the moving vehicle during driving. The driving right image can be obtained from a second preset database that has a storage relationship with the right lens. The second preset database stores multiple images captured by the right lens and the capture time of each image.

[0072] In at least one embodiment of the present application, the electronic device acquires a vehicle driving image of the vehicle during driving by one or a combination of the following methods:

[0073] (1) Controlling a binocular camera to capture the field of view of the moving vehicle during the moving process, and obtaining the moving left image and the moving right image.

[0074] The binocular camera can acquire the field of view image of the moving vehicle in real time during the moving process.

[0075] (2) Obtaining an image corresponding to the moving vehicle from a first preset database as the moving left image, obtaining a shooting time of the moving left image from the first preset database, and obtaining an image corresponding to both the moving vehicle and the shooting time from a second preset database as the moving right image.

[0076] Through the traveling vehicle and the shooting time, a traveling right image corresponding to the traveling left image can be accurately acquired.

[0077] This embodiment can obtain the vehicle driving image from multiple aspects, which is beneficial to the adjustment of the autoencoder.

[0078] S11 , performing instance segmentation processing on the vehicle driving image to obtain an instance segmented image, wherein the instance segmented image includes an instance segmented left image corresponding to the driving left image and an instance segmented right image corresponding to the driving right image.

[0079] In at least one embodiment of the present application, the instance segmented image includes label information of the vehicle driving image. It is understandable that when the vehicle driving image carries image information of a kitten, the instance segmented image carries frame information indicating the kitten.

[0080] In at least one embodiment of the present application, the electronic device performs instance segmentation processing on the vehicle driving image to obtain the instance segmented image, including:

[0081] Inputting the vehicle driving image into a pre-trained instance segmentation model to obtain an output image;

[0082] Detecting the segmentation accuracy of the instance segmentation model on the vehicle driving image based on the output image;

[0083] If the segmentation accuracy is greater than a preset threshold, the output image is determined as the instance segmented image.

[0084] The instance segmentation model includes a feature extraction network layer, a feature mapping layer, a classification layer, etc. The feature extraction network layer is used to extract feature information from the vehicle driving image, and the feature extraction network layer may include multiple convolutional layers. The feature mapping layer is used to map the feature information of the same subject into a feature map. For example, if the vehicle driving image carries a pedestrian subject and a kitten subject, the feature map obtained by the feature mapping layer includes a feature map corresponding to the pedestrian subject and a feature map corresponding to the kitten subject. The classification layer is used to identify the type corresponding to the feature map. The classification layer includes a fully connected layer.

[0085] The preset threshold value may be a value set according to actual needs. For example, the preset threshold value may be 95%.

[0086] The instance segmentation model can realize automatic instance segmentation of the vehicle driving image, thereby improving the generation efficiency of the output image. At the same time, by detecting the output image, the accuracy of the instance segmented image can be improved.

[0087] In this embodiment, the electronic device detecting, based on the output image, the segmentation accuracy of the instance segmentation model on the vehicle driving image includes:

[0088] Identifying a subject position where a subject tag is located in the vehicle driving image;

[0089] Obtaining a predicted position corresponding to the subject label from the output image;

[0090] The ratio of the position difference between the predicted position and the main body position at the main body position is calculated to obtain the segmentation accuracy.

[0091] The subject position and the predicted position are analyzed by using the same subject label, thereby improving the calculation accuracy of the segmentation accuracy.

[0092] S12: reconstructing the left driving image based on a pre-built autoencoder to obtain a predicted disparity map.

[0093] In at least one embodiment of the present application, the predicted disparity map includes a predicted disparity value for each pixel in the driving left image.

[0094] In at least one embodiment of the present application, the autoencoder includes an encoder and a decoder, and the electronic device reconstructs the driving left image based on the pre-built autoencoder to obtain a predicted disparity map including:

[0095] performing encoding processing on the traveling left image based on the encoder to obtain a feature vector of the traveling left image;

[0096] Mapping processing is performed on the feature vector based on the decoder to obtain the predicted disparity map.

[0097] Through the above implementation, the predicted disparity map can be quickly generated.

[0098] S13, generating a first error value of the autoencoder for the vehicle driving image based on the driving left image, the predicted disparity map and the driving right image, and generating a second error value of the autoencoder for the instance segmented image based on the instance segmented left image, the predicted disparity map and the instance segmented right image.

[0099] In at least one embodiment of the present application, the electronic device generating a first error value of the autoencoder for the vehicle driving image according to the driving left image, the predicted disparity map, and the driving right image includes:

[0100] Obtaining a predicted disparity value corresponding to each pixel in the driving left image from the predicted disparity map;

[0101] Locating a first pixel position of each pixel in the left driving image;

[0102] determining a second pixel position based on the predicted disparity value and the first pixel position;

[0103] Obtaining an RGB pixel value of a pixel point corresponding to the first pixel position from the left driving image;

[0104] Generate a predicted right image of the driving left image by the autoencoder according to the RGB pixel value of the pixel point corresponding to the first pixel position and the second pixel position;

[0105] Generating the first error value according to the driving right image and the predicted right image includes:

[0106]

[0107] Wherein, MSE refers to the first error value, y is the RGB pixel value of the right driving image, is the RGB pixel value of the predicted right image.

[0108] Among them, the predicted right image is an image generated by using the RGB pixel value at the first pixel position in the traveling left image as the RGB pixel value at the second pixel position. For example, the RGB pixel value of the pixel corresponding to the first pixel position (100, 200) in the traveling left image is x, and the predicted disparity value of the first pixel position (100, 200) in the traveling left image in the predicted disparity map is (20, 0). Then the second pixel position in the predicted right image is (120, 200), and the RGB pixel value of the pixel corresponding to the second pixel position (120, 200) is x.

[0109] Specifically, the electronic device determining the second pixel position based on the predicted disparity value and the first pixel position includes:

[0110] The second pixel position is obtained by calculating a sum of the predicted disparity value and the first pixel position.

[0111] For example, if the first pixel position is (100, 200), the predicted disparity value of the first pixel position (100, 200) in the predicted disparity map is (20, 0), and the second pixel position in the predicted right image is (120, 200).

[0112] The predicted right image can be accurately determined by the traveling left image and the predicted disparity map, and the first error value can be accurately determined by analyzing the pixel relationship between the traveling right image and the predicted right image.

[0113] In at least one embodiment of the present application, the manner in which the electronic device generates the second error value of the autoencoder for the instance segmented image based on the instance segmented left image, the predicted disparity map and the instance segmented right image is similar to the manner in which the electronic device generates the first error value of the autoencoder for the vehicle driving image based on the driving left image, the predicted disparity map and the driving right image, and this application will not elaborate on this.

[0114] S14: Adjust the autoencoder according to the first error value and the second error value to obtain an autoencoder model.

[0115] In at least one embodiment of the present application, the autoencoder model refers to an autoencoder when the total error value of training no longer decreases.

[0116] In at least one embodiment of the present application, the electronic device adjusts the autoencoder according to the first error value and the second error value to obtain the autoencoder model, including:

[0117] Calculating the sum of the first error value and the second error value to obtain a total training error value of the autoencoder;

[0118] The configuration parameters in the autoencoder are adjusted until the total training error value no longer decreases, thereby obtaining the autoencoder model.

[0119] Among them, the configuration parameters refer to the parameters of the initialization configuration in the decoder and the encoder. For example, the configuration parameters include the number of deconvolution layers in the decoder and the size of the convolution kernel in the deconvolution layer. The configuration parameters also include the number of convolution layers in the encoder and the size of the convolution kernel in the convolution layer.

[0120] The total training error value can be accurately determined by using the first error value and the second error value, thereby improving the accuracy of the autoencoder model.

[0121] S15 , obtaining a test driving image of the vehicle to be tested during driving, and reconstructing the test driving image according to the autoencoder model to obtain a target disparity map.

[0122] In at least one embodiment of the present application, the test driving image refers to an image corresponding to the field of view of the vehicle to be tested during driving.

[0123] The target disparity map includes the disparity value corresponding to each pixel in the test driving image.

[0124] In at least one embodiment of the present application, the electronic device reconstructs the test driving image according to the autoencoder model to obtain a target disparity map in a manner similar to the manner in which the electronic device reconstructs the driving left image based on a pre-built autoencoder to obtain a predicted disparity map, and this application will not elaborate on this.

[0125] S16: Convert the target disparity map to obtain a depth image corresponding to the test driving image.

[0126] In at least one embodiment of the present application, the depth image refers to a depth map corresponding to the field of view of the vehicle under test during driving, and the depth image includes distance values ​​from the camera of the test driving image to objects in the field of view.

[0127] In at least one embodiment of the present application, the electronic device converting the target disparity map to obtain a depth image corresponding to the test driving image includes:

[0128] Obtaining a focal length value of a shooting device for the test driving image, and obtaining a lens distance between lenses in the shooting device;

[0129] Acquire a target disparity value of each pixel in the test driving image from the target disparity map;

[0130] The depth pixel value corresponding to each pixel point in the test driving image is calculated according to the focal length value, the lens distance and the target parallax value. The calculation formula of the depth pixel value is:

[0131]

[0132] Wherein, z refers to the depth pixel value, b refers to the lens distance, f refers to the focal length value, and d refers to the target disparity value;

[0133] The depth image is generated according to each pixel point in the test driving image and the depth pixel value.

[0134] Through the above implementation, the depth image can be accurately generated, which is beneficial to controlling the automatic driving of the vehicle to be tested.

[0135] It can be seen from the above technical solution that the present application can accurately identify the label information in the vehicle driving image by performing instance segmentation processing on the vehicle driving image, and analyze the first error value of the autoencoder for the vehicle driving image and the second error value of the autoencoder for the instance segmented image through the predicted disparity map generated by encoding the driving left image, thereby enhancing the consistency of the predicted disparity map on the same object, and then eliminating the interference of the autoencoder on image encoding, and improving the accuracy of the first error value and the second error value. By adjusting the autoencoder in combination with the first error value and the second error value, the image processing capability of the autoencoder model can be further improved, so that the test driving image can be processed based on the autoencoder model, and the accuracy and fineness of the depth image can be improved, thereby improving the accuracy of autonomous driving.

[0136] like Figure 3 As shown, Figure 3 1 is a functional block diagram of a preferred embodiment of the image processing device of the present application. The image processing device 11 includes an acquisition unit 110, a segmentation unit 111, a reconstruction unit 112, a generation unit 113, an adjustment unit 114, and a conversion unit 115. The modules / units referred to herein are a series of computer-readable instruction segments that can be acquired by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0137] Specifically, the acquisition unit can be used to acquire a vehicle driving image of a moving vehicle during driving, wherein the vehicle driving image includes a driving left image and a driving right image. The segmentation unit can be used to perform instance segmentation processing on the vehicle driving image to obtain an instance segmented image, wherein the instance segmented image includes an instance segmented left image corresponding to the driving left image and an instance segmented right image corresponding to the driving right image. The reconstruction unit can be used to reconstruct the driving left image based on a pre-built autoencoder to obtain a predicted disparity map. The generation unit can be used to generate the driving left image, the predicted disparity map and the driving right image according to the driving left image. The autoencoder generates a first error value of the vehicle driving image, and generates a second error value of the autoencoder for the instance segmented image based on the instance segmented left image, the predicted disparity map and the instance segmented right image. The adjustment unit can be used to adjust the autoencoder according to the first error value and the second error value to obtain an autoencoder model. The reconstruction unit can also be used to obtain a test driving image of the vehicle to be tested during driving, and reconstruct the test driving image according to the autoencoder model to obtain a target disparity map. The conversion unit can be used to convert the target disparity map to obtain a depth image corresponding to the test driving image.

[0138] The present application can accurately identify the label information in the vehicle driving image by performing instance segmentation processing on the vehicle driving image, and analyze the first error value of the autoencoder for the vehicle driving image and the second error value of the autoencoder for the instance segmented image through the predicted disparity map generated by encoding the left driving image, thereby enhancing the consistency of the predicted disparity map on the same object, thereby eliminating the interference of the autoencoder on image encoding, and improving the accuracy of the first error value and the second error value. By adjusting the autoencoder in combination with the first error value and the second error value, the image processing capability of the autoencoder model can be further improved, so that the test driving image can be processed based on the autoencoder model, and the accuracy and fineness of the depth image can be improved, thereby improving the accuracy of autonomous driving.

[0139] like Figure 4 , which is a structural diagram of an electronic device for implementing a preferred embodiment of the image processing method of the present application.

[0140] In one embodiment of the present application, the electronic device 1 includes, but is not limited to, a memory 12 , a processor 13 , and a computer program stored in the memory 12 and executable on the processor 13 , such as an image processing program.

[0141] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 1 may also include input and output devices, network access devices, buses, etc.

[0142] The processor 13 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 13 is the computing core and control center of the electronic device 1, and utilizes various interfaces and lines to connect various parts of the entire electronic device 1, and obtain the operating system of the electronic device 1 and various installed applications, program codes, etc.

[0143] The processor 13 obtains the operating system and various installed applications of the electronic device 1. The processor 13 obtains the applications to implement the steps in the above-mentioned various image processing method embodiments, for example Figure 2 Steps shown.

[0144] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and retrieved by the processor 13 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the acquisition process of the computer program in the electronic device 1.

[0145] The memory 12 can be used to store the computer programs and / or modules. The processor 13 implements the various functions of the electronic device 1 by running or obtaining the computer programs and / or modules stored in the memory 12, and calling the data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), and the data storage area may store data created based on the use of the electronic device. In addition, the memory 12 may include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device.

[0146] The memory 12 may be an external memory and / or an internal memory of the electronic device 1. Furthermore, the memory 12 may be a physical memory, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0147] If the modules / units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is obtained by the processor, it can implement the steps of each of the above-mentioned method embodiments.

[0148] The computer program includes computer program code, which may be in source code form, object code form, an accessible file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0149] Combine Figure 2 , the memory 12 in the electronic device 1 stores a plurality of instructions to implement an image processing method, and the processor 13 can obtain the plurality of instructions to implement: obtaining a vehicle driving image of a moving vehicle during driving, the vehicle driving image including a driving left image and a driving right image; performing instance segmentation processing on the vehicle driving image to obtain an instance segmentation image, the instance segmentation image including an instance segmentation left image corresponding to the driving left image and an instance segmentation right image corresponding to the driving right image; reconstructing the driving left image based on a pre-built autoencoder to obtain a predicted disparity map; and The method comprises the steps of: generating a first error value of the autoencoder for the vehicle driving image based on the left driving image, the predicted disparity map, and the right driving image, and generating a second error value of the autoencoder for the instance segmented image based on the instance segmented left image, the predicted disparity map, and the instance segmented right image; adjusting the autoencoder based on the first error value and the second error value to obtain an autoencoder model; obtaining a test driving image of the vehicle to be tested during driving, and reconstructing the test driving image based on the autoencoder model to obtain a target disparity map; and converting the target disparity map to obtain a depth image corresponding to the test driving image.

[0150] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0152] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0153] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0154] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0155] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in this application may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Acquire a vehicle driving image of the vehicle during driving, wherein the vehicle driving image includes a left driving image and a right driving image; performing instance segmentation processing on the vehicle driving image to obtain an instance segmented image, wherein the instance segmented image includes an instance segmented left image corresponding to the driving left image and an instance segmented right image corresponding to the driving right image; Reconstructing the left driving image based on a pre-built autoencoder to obtain a predicted disparity map; generating a first error value of the autoencoder for the vehicle driving image according to the driving left image, the predicted disparity map, and the driving right image, and generating a second error value of the autoencoder for the instance segmented image according to the instance segmented left image, the predicted disparity map, and the instance segmented right image; Adjust the autoencoder according to the first error value and the second error value to obtain an autoencoder model; Acquire a test driving image of the vehicle to be tested during driving, and reconstruct the test driving image according to the autoencoder model to obtain a target disparity map; The target disparity map is converted to obtain a depth image corresponding to the test driving image.

2. The image processing method according to claim 1, wherein: The method of obtaining the vehicle driving image during the driving process includes one or more of the following methods: Controlling a binocular camera to capture the field of view of the moving vehicle during the moving process to obtain the moving left image and the moving right image; An image corresponding to the moving vehicle is obtained from a first preset database as the moving left image, a shooting time of the moving left image is obtained from the first preset database, and an image corresponding to both the moving vehicle and the shooting time is obtained from a second preset database as the moving right image.

3. The image processing method according to claim 1, wherein: The performing instance segmentation processing on the vehicle driving image to obtain the instance segmented image includes: Inputting the vehicle driving image into a pre-trained instance segmentation model to obtain an output image; Detecting the segmentation accuracy of the instance segmentation model on the vehicle driving image based on the output image; If the segmentation accuracy is greater than a preset threshold, the output image is determined as the instance segmented image.

4. The image processing method according to claim 1, wherein: The autoencoder includes an encoder and a decoder, and the reconstructing process of the driving left image based on the pre-built autoencoder to obtain the predicted disparity map includes: performing encoding processing on the traveling left image based on the encoder to obtain a feature vector of the traveling left image; Mapping processing is performed on the feature vector based on the decoder to obtain the predicted disparity map.

5. The image processing method according to claim 1, wherein: Generating a first error value of the autoencoder for the vehicle driving image according to the driving left image, the predicted disparity map, and the driving right image includes: Obtaining a predicted disparity value corresponding to each pixel in the driving left image from the predicted disparity map; Locating a first pixel position of each pixel in the left driving image; determining a second pixel position based on the predicted disparity value and the first pixel position; Obtaining an RGB pixel value of a pixel point corresponding to the first pixel position from the left driving image; Generate a predicted right image of the driving left image by the autoencoder according to the RGB pixel value of the pixel point corresponding to the first pixel position and the second pixel position; Generating the first error value according to the driving right image and the predicted right image includes: Wherein, MSE refers to the first error value, y is the RGB pixel value of the right driving image, is the RGB pixel value of the predicted right image.

6. The image processing method according to claim 1, wherein: The adjusting the autoencoder according to the first error value and the second error value to obtain the autoencoder model includes: Calculating the sum of the first error value and the second error value to obtain a total training error value of the autoencoder; The configuration parameters in the autoencoder are adjusted until the total training error value no longer decreases, thereby obtaining the autoencoder model.

7. The image processing method according to claim 1, wherein: Converting the target disparity map to obtain a depth image corresponding to the test driving image includes: Obtaining a focal length value of a shooting device for the test driving image, and obtaining a lens distance between lenses in the shooting device; Acquire a target disparity value of each pixel in the test driving image from the target disparity map; The depth pixel value corresponding to each pixel point in the test driving image is calculated according to the focal length value, the lens distance and the target parallax value. The calculation formula of the depth pixel value is: Wherein, z refers to the depth pixel value, b refers to the lens distance, f refers to the focal length value, and d refers to the target disparity value; The depth image is generated according to each pixel point in the test driving image and the depth pixel value.

8. An image processing device, characterized in that: The image processing device comprises: an acquisition unit, configured to acquire a vehicle driving image of the vehicle during driving, wherein the vehicle driving image includes a left driving image and a right driving image; a segmentation unit, configured to perform instance segmentation processing on the vehicle driving image to obtain instance segmented images, wherein the instance segmented images include an instance segmented left image corresponding to the vehicle driving left image and an instance segmented right image corresponding to the vehicle driving right image; A reconstruction unit, configured to reconstruct the left driving image based on a pre-built autoencoder to obtain a predicted disparity map; a generating unit, configured to generate a first error value of the autoencoder for the vehicle driving image based on the driving left image, the predicted disparity map, and the driving right image, and to generate a second error value of the autoencoder for the instance segmented image based on the instance segmented left image, the predicted disparity map, and the instance segmented right image; an adjusting unit, configured to adjust the autoencoder according to the first error value and the second error value to obtain an autoencoder model; The reconstruction unit is further configured to obtain a test driving image of the vehicle under test during driving, and reconstruct the test driving image according to the autoencoder model to obtain a target disparity map; A conversion unit is used to convert the target disparity map to obtain a depth image corresponding to the test driving image.

9. An electronic device, characterized in that: The electronic device comprises: a memory storing at least one instruction; and A processor executes the at least one instruction to implement the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the image processing method according to any one of claims 1 to 7.

Images