Model training method, image processing method, and related apparatus
The image processing model is trained using the optical flow estimation results of the trained model through a self-supervised method, which solves the accuracy problem of pixel depth prediction in the image and improves the training efficiency and accuracy of the model.
Patent Information
- Application Number
- PCT/CN2024/091359
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-05-07
- Publication Date
- 2024-12-05
AI Technical Summary
In the field of artificial intelligence, when using machine learning models to predict the depth of pixel points in an image, accuracy is difficult to obtain, resulting in low model training efficiency and accuracy.
The model is trained in a self-supervised manner, and the loss function is calculated by utilizing the first optical flow estimation result and the second optical flow estimation result generated by the trained second machine learning model, and knowledge distillation is carried out to improve the training accuracy of the first machine learning model. Spend.
Improve the accuracy of the model output results, reduce additional errors, and enhance the training model performance.
Smart Images

Figure CN2024091359_05122024_PF_FP_ABST
Abstract
Description
A model training method, image processing method and related equipment
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 29, 2023, with application number 202310621754.8 and invention name “A model training method, image processing method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of artificial intelligence, and in particular to a model training method, an image processing method, and related equipment. Background Art
[0003] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.
[0004] Using machine learning models to predict the depth of pixels in an image is an application of artificial intelligence. However, since the accurate depth of pixels in an image is difficult to obtain, a solution for training the machine learning model using self-supervision is urgently needed.
[0005] Summary of the Invention
[0006] An embodiment of the present application provides a model training method, an image processing method, and related equipment. The loss function is obtained based on a first optical flow estimation result and a second optical flow estimation result. The first optical flow estimation result is generated by a second machine learning model that has performed a training operation. The accuracy of the first optical flow estimation result is relatively high. That is, the process of training the first machine learning model using the aforementioned loss function introduces the idea of knowledge distillation, and provides a solution for training the first machine learning model in a self-supervised manner.
[0007] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:
[0008] In a first aspect, embodiments of the present application provide a model training method that can be used in the field of image processing in the field of artificial intelligence. The method includes: a training device inputs a first image into a first machine learning model to obtain first information output by the first machine learning model, wherein the first information indicates the depth of pixels in the first image. The training device inputs the first image and the second image into a second machine learning model, and generates an optical flow from the first image to the second image using the second machine learning model to obtain a first optical flow estimation result, wherein the second machine learning model is a machine learning model that has been trained, and the first optical flow estimation result includes first movement parameters of pixels in the first image from the first image to the second image. The training device generates a second optical flow estimation result based on the first information, wherein the second optical flow estimation result includes second movement parameters of pixels in the first image from the first image to the second image. The training device determines a function value of a loss function based on the first and second optical flow estimation results, and trains the first machine learning model based on the function value of the loss function. Exemplarily, the parameters of the second machine learning model are maintained unchanged, and the parameters of the first machine learning model are updated based on the function value of the loss function.
[0009] In this implementation, the loss function is obtained based on the first optical flow estimation result and the second optical flow estimation result. The first optical flow estimation result is generated by the second machine learning model that has performed the training operation. The accuracy of the first optical flow estimation result is relatively high. That is, the process of training the first machine learning model using the aforementioned loss function introduces the idea of knowledge distillation, and provides a solution for training the first machine learning model in a self-supervised manner. In addition, the optical flow estimation result is generated based on the depth information output by the first machine learning model. The execution process of the aforementioned steps is less affected by factors such as the different light when the first image and the second image are collected. That is, the accuracy of the aforementioned steps is relatively high, thereby reducing the additional error introduced by the step of "generating the second optical flow estimation result based on the first information", which is conducive to improving the accuracy of the results output by the trained first machine learning model.
[0010] In one possible implementation, the method also includes: the training device inputs the second image and the first image into a second machine learning model, generates an optical flow from the second image to the first image through the second machine learning model, and obtains a third optical flow estimation result, and the third optical flow estimation result includes a third movement parameter of a pixel point in the second image from the second image to the first image; wherein, the loss function indicates the similarity between the first movement parameter and the second movement parameter corresponding to each first pixel point in the first image, the multiple first pixel points in the first image are screened out from the multiple pixel points included in the first image, the confidence of the first movement parameter of each first pixel point meets the first condition, and the confidence of the first movement parameter of the first pixel point is determined based on the first optical flow estimation result and the third optical flow estimation result.
[0011] In this implementation, the loss function indicates the similarity between the first movement parameter and the second movement parameter of the first pixel point, that is, the first movement parameters of multiple first pixel points in the first optical flow estimation result are regarded as the output of the teacher model, and the first movement parameters of multiple first pixel points are used as supervision items in the training process; and the first pixel points are screened out from the multiple pixel points included in the first image, and the confidence of the first movement parameter of each first pixel point meets the first condition, that is, the accuracy of the first movement parameter of each first pixel point is higher, then the accuracy of the supervision item used in the training process is higher, which is conducive to improving the accuracy of the information output by the first machine learning model after training.
[0012] In one possible implementation, the loss function indicates the similarity between a first movement parameter and a second movement parameter corresponding to each first pixel point in the first image, where the multiple first pixel points in the first image are screened out from the multiple pixel points included in the first image, and the depth of each first pixel point is less than or equal to a depth threshold.
[0013] In this implementation, when the depth of a certain pixel point in the first image is too large, the error in predicting the movement parameters of the aforementioned pixel point from the first image to the second image is likely to be larger. The depth of each first pixel point is less than or equal to the depth threshold, so the accuracy of the first movement parameter of each first pixel point is higher, that is, the accuracy of the supervision item used in the training process is higher, which is conducive to improving the accuracy of the information output by the first machine learning model after training.
[0014] In one possible implementation, the method further includes: a training device determining, based on the first optical flow estimation result, the focal length of the first camera, and position information, pixels in the first image whose depth is less than or equal to a depth threshold, wherein the position information indicates a relative position between a first position and a second position, the first position being the shooting position of the first image, and the second position being the shooting position of the second image. In this implementation, since the first optical flow estimation result is generated by a first machine learning model that has been trained, i.e., the first optical flow estimation result has a high accuracy, using the first optical flow estimation result to perform the operation of "screening out the first pixel in the first image whose depth is less than or equal to the depth threshold" is beneficial to improving the accuracy of the aforementioned operation.
[0015] In one possible implementation, the loss function indicates the similarity between a first movement parameter and a second movement parameter corresponding to each first pixel point in the first image, the multiple first pixel points in the first image are screened out from the multiple pixel points included in the first image, the confidence of the first movement parameter of each first pixel point satisfies the first condition, and the depth of each first pixel point is less than or equal to the depth threshold.
[0016] In one possible implementation, the training device generates a second optical flow estimation result based on the first information, including: the training device generates a second optical flow estimation result based on the first information and position information, the position information indicates the relative position between the first position and the second position, the first position is the shooting position of the first image, and the second position is the shooting position of the second image.
[0017] In this implementation, since the first information indicates the depth of multiple pixels in the first image, after understanding the relative position between the first position (i.e., the shooting position of the first image) and the second position (i.e., the shooting position of the second image), it is possible to infer the movement parameters of the pixel points in the first image from the first image to the second image, thereby providing a feasible solution for calculating the second optical flow estimation result based on the first information; optionally, if the aforementioned reasoning process does not use a machine learning model, it is beneficial to improve the certainty of the aforementioned reasoning process, so as to reduce the possibility of introducing errors in the aforementioned reasoning process, that is, to reduce the possibility of introducing errors in the training process of the first machine learning model, which is beneficial to improve the accuracy of the information output by the trained first machine learning model.
[0018] In one possible implementation, the first image is captured at a first position, and the second image is captured at a second position, where the first position and the second position are different. If the first image and the second image are from the same camera, then the first camera (i.e., the second camera) captures the first image and the second image at two different positions, respectively. Alternatively, the first image is from two different cameras, illustratively, the first camera and the second camera may be binocular cameras. In this implementation, multiple methods for determining the sources of the first image and the second image are provided, which helps to improve the implementation flexibility of this solution.
[0019] On the second aspect, an embodiment of the present application provides an image processing method that can be used in the field of image processing in the field of artificial intelligence. The method includes: executing a device to obtain an image to be processed, inputting the image to be processed into a first machine learning model, and obtaining second information output by the first machine learning model. The second information indicates the depth of the pixel points in the image to be processed. The specific expression form of the "second information" is similar to the specific expression form of the "first information", and will not be repeated here.
[0020] Among them, the first machine learning model is trained using a loss function, and the loss function is obtained based on the first optical flow estimation result and the second optical flow estimation result. The first optical flow estimation result includes a first movement parameter of a pixel point in the first image from the first image to the second image, and the second optical flow estimation result includes a second movement parameter of a pixel point in the first image from the first image to the second image; the first optical flow estimation result is generated by a second machine learning model that has performed a training operation, and the second optical flow estimation result is obtained based on first information indicating the depth of a pixel point in the first image. When the output of the first machine learning model is the first information, the input of the first machine learning model is the first image.
[0021] In the second aspect of this application, the specific implementation methods of the steps in the second aspect, the meanings of the terms and the beneficial effects brought about can all be referred to the first aspect and will not be repeated here.
[0022] In a third aspect, an embodiment of the present application provides a model training device that can be used in the field of image processing in the field of artificial intelligence, and the device includes: an input module for inputting a first image into a first machine learning model to obtain first information output by the first machine learning model, where the first information indicates the depth of a pixel in the first image; the input module is also used to input the first image and the second image into a second machine learning model, and generate a first optical flow estimation result through the second machine learning model, wherein the second machine learning model is a machine learning model that has performed a training operation, and the first optical flow estimation result includes a first movement parameter of a pixel in the first image from the first image to the second image; a generation module is used to generate a second optical flow estimation result based on the first information, and the second optical flow estimation result includes a second movement parameter of a pixel in the first image from the first image to the second image; a training module is used to determine a function value of a loss function based on the first optical flow estimation result and the second optical flow estimation result, and train the first machine learning model based on the function value of the loss function.
[0023] In the third aspect of this application, the model training device is also used to execute the steps performed by the training device in the first aspect and various possible implementation methods of the first aspect. The specific implementation methods of the steps in the third aspect, the meanings of the terms and the beneficial effects brought about can all be referred to the first aspect and will not be repeated here.
[0024] In a fourth aspect, an embodiment of the present application provides an image processing device, which can be used in the field of image processing in the field of artificial intelligence, and the device includes: an acquisition module for acquiring an image to be processed; an input module for inputting the image to be processed into a first machine learning model to obtain second information output by the first machine learning model, and the second information indicates the depth of the pixel points in the image to be processed; wherein the first machine learning model is trained using a loss function, and the loss function is obtained based on the first optical flow estimation result and the second optical flow estimation result, the first optical flow estimation result includes the first movement parameter of the pixel points in the first image from the first image to the second image, and the second optical flow estimation result includes the second movement parameter of the pixel points in the first image from the first image to the second image; the first optical flow estimation result is generated by a second machine learning model that has performed a training operation, and the second optical flow estimation result is obtained based on the first information indicating the depth of the pixel points in the first image. When the output of the first machine learning model is the first information, the input of the first machine learning model is the first image.
[0025] In the fourth aspect of this application, the specific implementation methods of the steps in the fourth aspect, the meanings of the terms and the beneficial effects brought about can all be referred to the second aspect and will not be repeated here.
[0026] In the fifth aspect, an embodiment of the present application provides an execution device, including a processor and a memory, the processor is coupled to the memory, the memory is used to store programs; the processor is used to execute the programs in the memory, so that the execution device executes the image processing method described in the second aspect above.
[0027] In the sixth aspect, an embodiment of the present application provides a training device, including a processor and a memory, where the processor is coupled to the memory, the memory is used to store programs; and the processor is used to execute the programs in the memory, so that the training device executes the training method of the model described in the first aspect above.
[0028] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the first or second aspect above.
[0029] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes a program. When the program runs on a computer, it enables the computer to execute the method described in the first or second aspect above.
[0030] In a ninth aspect, the present application provides a chip system, which includes a processor for supporting the implementation of the functions involved in the above aspects, for example, sending or processing the data and / or information involved in the above methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the terminal device or communication device. The chip system can be composed of a chip or can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] FIG1 is a schematic diagram of the structure of an artificial intelligence main framework provided in an embodiment of the present application;
[0032] FIG2 is a system architecture diagram of an image processing system provided in an embodiment of the present application;
[0033] FIG3 is a schematic diagram of a training method for a model provided in an embodiment of the present application;
[0034] FIG4 is another flow chart of a method for training a model according to an embodiment of the present application;
[0035] FIG5 is a schematic diagram of “x-axis and y-axis in a camera coordinate system” and “u-axis and v-axis in an image coordinate system” provided by an embodiment of the present application;
[0036] FIG6 is a schematic diagram of a process for generating a second optical flow estimation result according to an embodiment of the present application;
[0037] FIG7 is a schematic diagram of a set consisting of all first pixel points included in a first image provided by an embodiment of the present application;
[0038] FIG8 is another flow chart of a method for training a model according to an embodiment of the present application;
[0039] FIG9 is a flow chart of an image processing method provided in an embodiment of the present application;
[0040] FIG10 is a schematic structural diagram of a model training device provided in an embodiment of the present application;
[0041] FIG11 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;
[0042] FIG12 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application;
[0043] FIG13 is a schematic diagram of a structure of a training device provided in an embodiment of the present application;
[0044] FIG14 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0046] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0047] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the AI framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.
[0048] (1) Infrastructure
[0049] The infrastructure provides computing power for AI systems, enabling communication with the outside world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by smart chips, which can specifically adopt hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs); the basic platform includes related platform guarantees and support such as distributed computing frameworks and networks, and can include cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to obtain data, which is then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.
[0050] (2) Data
[0051] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.
[0052] (3) Data processing
[0053] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.
[0054] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0055] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0056] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.
[0057] (4) General ability
[0058] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0059] (5) Smart products and industry applications
[0060] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, smart cities, etc.
[0061] The method provided in this application can be applied to various application fields of artificial intelligence technology. Specifically, it can be used to generate the depth of pixel points in an image in various application scenarios. Since the method provided in this application can be used in fields such as autonomous driving, smart terminals, and smart manufacturing, the following examples are given of application scenarios in multiple application fields of this application.
[0062] Application area 1: autonomous driving
[0063] For example, an autonomous vehicle can capture images of its surroundings using a camera. After performing object detection on the images, the autonomous vehicle can determine the category of at least one object surrounding the vehicle and the location of each object in the image. Furthermore, the autonomous vehicle can generate the depth of pixels in the images using a first machine learning model, thereby determining the distance between the at least one object surrounding the vehicle and the vehicle. The autonomous vehicle can then plan its driving path based on the aforementioned information. The autonomous vehicle can be a car, truck, motorcycle, bus, boat, airplane, helicopter, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, or train, among others, and is not specifically limited in the present embodiments.
[0064] Application field 2: smart terminal field
[0065] For example, if the smart terminal is an augmented reality (AR) device, and the AR device superimposes the generated virtual objects on the real-world scene, the AR device can first obtain an image of the surrounding environment, and use the first machine learning model to generate the depth of the pixel points in the image of the surrounding environment. If the virtual object can be three-dimensionally reconstructed based on the depth of the pixel points in the image of the surrounding environment, the virtual object can be better superimposed on the surrounding environment.
[0066] For another example, the smart terminal is a drone. The drone can collect images of the surrounding environment. By using the first machine learning model to generate the depth of pixel points in the image of the surrounding environment, the drone can understand the distance between the surrounding objects and the mobile robot, and can avoid obstacles as much as possible when planning the path of the drone.
[0067] Application field 3: intelligent manufacturing
[0068] For example, mobile robots may be used in the field of intelligent manufacturing. The mobile robots can collect images of the surrounding environment. By using a first machine learning model to generate the depth of pixel points in the images of the surrounding environment, obstacles can be avoided as much as possible when planning the path of the mobile robot.
[0069] It should be noted that the method provided in this application can also be applied to other application scenarios. The above examples of various application scenarios of this application are only for the convenience of understanding this solution and are not used to limit this solution.
[0070] In all the above application scenarios, it is necessary to use the first machine learning model to predict the depth of the pixels in the image. However, since the accurate depth of the pixels in the image is difficult to obtain, in order to solve the above problem, the method provided in this application uses a self-supervised method to train the first machine learning model. Before describing the method provided in this application in detail, please refer to Figure 2. Figure 2 is a system architecture diagram of the image processing system provided in an embodiment of the present application. In Figure 2, the image processing system 200 includes a training device 210, a database 220, an execution device 230 and a data storage system 240. The execution device 230 includes a computing module 231.
[0071] Among them, the database 220 stores a training data set. During the training phase of the first machine learning model 201, the training device 210 generates the first machine learning model 201 and iteratively trains the first machine learning model 201 using the training data set to obtain the first machine learning model 201 that has been trained. The first machine learning model 201 can be specifically expressed as a neural network or a non-neural network model. In the embodiments of the present application, only the first machine learning model 201 expressed as a neural network is used as an example for description.
[0072] Specifically, please refer to Figure 3, which is a schematic diagram of a model training method provided in an embodiment of the present application. In particular, 301, the training device 210 inputs a first image into the first machine learning model 201, and obtains first information output by the first machine learning model 201, where the first information indicates the depth of pixels in the first image.
[0073] 302. The training device 210 inputs the first image and the second image into a second machine learning model, and generates a first optical flow estimation result through the second machine learning model, wherein the second machine learning model is a machine learning model that has performed a training operation, and the first optical flow estimation result includes a first movement parameter of a pixel point in the first image from the first image to the second image.
[0074] 303. The training device 210 generates a second optical flow estimation result based on the first information. The second optical flow estimation result includes a second movement parameter of a pixel point in the first image when moving from the first image to the second image.
[0075] 304. The training device 210 determines a function value of the loss function based on the first optical flow estimation result and the second optical flow estimation result, and trains the first machine learning model 210 based on the function value of the loss function.
[0076] In an embodiment of the present application, since the first optical flow estimation result is generated by the second machine learning model that has performed the training operation, the accuracy of the first optical flow estimation result is relatively high, that is, the process of training the first machine learning model using the aforementioned loss function introduces the idea of knowledge distillation, and provides a solution for training the first machine learning model in a self-supervised manner; in addition, the optical flow estimation result is generated based on the depth information output by the first machine learning model, and the execution process of the aforementioned steps is less affected by factors such as the different light when the first image and the second image are collected, that is, the accuracy of the aforementioned steps is relatively high, thereby reducing the additional error introduced by the step of "generating the second optical flow estimation result based on the first information", which is conducive to improving the accuracy of the results output by the trained first machine learning model.
[0077] The first machine learning model 201 obtained by the training device 210 after executing the training operation can be deployed to the computing module 231 of the execution device 230. For example, the execution device 230 can be a vehicle, a mobile phone, a tablet, an AR device, a virtual reality (VR) device, a drone, a robot, or other types of devices. The execution device 230 can call data, code, etc. in the data storage system 240, or it can store data, instructions, etc. in the data storage system 240. The data storage system 240 can be placed in the execution device 230, or the data storage system 240 can be an external memory relative to the execution device 230.
[0078] During the application stage of the first machine learning model 201, after the execution device 230 inputs the image to be processed into the first machine learning model 201 in the computing module 231, the second information output by the first machine learning model 201 can be obtained, and the second information indicates the depth of the pixel points in the image to be processed.
[0079] In some embodiments of the present application, referring to FIG. 2 , the execution device 230 and the client device may be integrated into the same device, and the user may interact directly with the execution device 230. For example, when the client device is a mobile phone or a tablet, the execution device 230 may be a module in the host processor (Host CPU) of the mobile phone or tablet that performs data processing using the first machine learning model. The execution device 230 may also be a graphics processing unit (GPU) or a neural network processor (NPU) in the mobile phone or tablet. The GPU or NPU is mounted on the host processor as a coprocessor, and the host processor assigns tasks.
[0080] It is worth noting that FIG2 is only a schematic diagram of the architecture of two data processing systems provided by an embodiment of the present invention, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in other embodiments of the present application, the execution device 230 and the client device can be independent devices, and the execution device 230 is configured with an input / output (I / O) interface to interact with the client device for data. After determining the image to be processed, the client device sends the image to be processed to the execution device 230 through the I / O interface. After the execution device 230 generates the second information corresponding to the image to be processed through the first machine learning model 201 in the computing module 231, the aforementioned second information can be returned to the client device through the I / O interface.
[0081] In combination with the above description, the following describes the specific implementation process of the training phase and application phase of the method provided in the embodiment of the present application.
[0082] 1. Training Phase
[0083] In the embodiment of the present application, the training phase describes the process of training the first machine learning model 201 by the training device 210 using the training data set in the database 220. Specifically, please refer to FIG4 , which is another flow chart of the model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application may include:
[0084] 401. Input a first image into a first machine learning model to obtain first information output by the first machine learning model, where the first information indicates the depth of a pixel in the first image.
[0085] In an embodiment of the present application, the training device can obtain training data from a training data set, and each training data includes a first image. The first image comes from a first camera, that is, the first image is taken by the first camera; the shooting position of the first image is the first position.
[0086] After the training device obtains the first image from the first training data (i.e., any training data in the training data set), the first image can be input into the first machine learning model, and the first image can be processed by the first machine learning model to obtain first information output by the first machine learning model, where the first information indicates the depth of one or more pixels in the first image. Exemplarily, the first information may include at least one set of sub-information corresponding one-to-one to at least one pixel in the first image, each set of sub-information including a pixel in the first image and the depth of the pixel.
[0087] 402. Input the first image and the second image into a second machine learning model, and generate a first optical flow estimation result through the second machine learning model, wherein the second machine learning model is a machine learning model that has performed a training operation, and the first optical flow estimation result includes a first movement parameter of a pixel point in the first image when moving from the first image to the second image.
[0088] In the embodiments of the present application, each training data set may further include a second image. For example, because the optical flow from the first image to the second image needs to be calculated, the ratio of the repeated image content between the first image and the second image in each training data set to the total image content of the first image may exceed a preset ratio. For example, the preset ratio may be 40%, 50%, 60%, or other values, and is not limited in the embodiments of the present application. It should be understood that this description is merely for facilitating understanding of the "relationship between the first image and the second image in a training data set" and does not imply that the training device needs to calculate the ratio of the repeated image content between the first image and the second image in each training data set to the total image content of the first image.
[0089] The second image originates from the second camera, i.e., the second image is captured by the second camera; the first camera and the second camera can be different cameras, or the second camera and the first camera can be the same camera. In the embodiments of the present application, multiple methods for determining the sources of the first image and the second image are provided, which helps to increase the flexibility of the implementation of the present solution.
[0090] The second image was captured at the second position, and the first position and the second position are different. If the first camera and the second camera are different, then the position of the first camera when capturing the first image is different from the position of the second camera when capturing the second image. For example, the first camera and the second camera may be binocular cameras. Alternatively, to ensure that the ratio of the repeated image content between the first image and the second image to the total image content of the first image exceeds a preset ratio, the distance between the first position and the second position is less than a distance threshold. Alternatively, if the first camera and the second camera are the same, then the first camera (also known as the second camera) captured the first image and the second image at two different positions.
[0091] After the training device obtains the first image and the second image from the first training data, the first image and the second image can be input into the second machine learning model, and the optical flow from the first image to the second image can be generated by the second machine learning model to obtain the first optical flow estimation result output by the second machine learning model.
[0092] Among them, the second machine learning model is a machine learning model that has performed training operations, that is, the first optical flow estimation result output by the second machine learning model can be considered as a prediction result with higher accuracy, and the second machine learning model can be understood as a "teacher model".
[0093] The first optical flow estimation result includes a first movement parameter of a pixel point in the first image when it moves from the first image to the second image. Exemplarily, the first image and the second image are both located in an image coordinate system, the image coordinate system includes two directions, u and v, and the unit in the image coordinate system is one pixel. The first optical flow result may include multiple first movement parameters corresponding one-to-one to multiple pixel points in the first image, each first movement parameter may be expressed as a first vector, and each first vector may include two numerical values, and the two numerical values in each first vector represent the amount of movement of a pixel point in the first image in the aforementioned two directions, u and v, when it moves from the first image to the second image.
[0094] For example, if the first movement parameter corresponding to a pixel point in the first image is (3, 5), it means that the pixel point moves 3 pixels in the positive direction of u and 5 pixels in the positive direction of v from the first image to the second image; for another example, if the first movement parameter corresponding to a pixel point in the first image is (-2, 0), it means that the pixel point moves 2 pixels in the negative direction of u from the first image to the second image, and does not move in the v direction, etc. It should be understood that the examples here are only for the convenience of understanding this scheme and are not used to limit this scheme.
[0095] 403. Input the second image and the first image into a second machine learning model, and generate a third optical flow estimation result through the second machine learning model, where the third optical flow estimation result includes a third movement parameter of a pixel point in the second image when moving from the second image to the first image.
[0096] In the embodiment of the present application, step 403 is an optional step. After the training device obtains the first image and the second image from the first training data, it can also input the second image and the first image into the second machine learning model, and generate the optical flow from the second image to the first image through the second machine learning model to obtain the third optical flow estimation result output by the second machine learning model. The third optical flow estimation result includes the third movement parameter of the pixel point in the second image when it moves from the second image to the first image. Exemplarily, the third optical flow estimation result may include multiple third movement parameters corresponding to multiple pixels in the second image, each third movement parameter may be expressed as a third vector, and each third vector may include two numerical values. The two numerical values in each third vector represent the movement amount of a pixel point in the second image in the above-mentioned u and v directions when it moves from the second image to the first image. The specific expression form of the "third vector" is similar to the specific expression form of the "first vector", and no example is given here. It should be noted that the concept of the "second vector" will be used in subsequent steps and will not be described here.
[0097] 404. Generate a second optical flow estimation result based on the first information, where the second optical flow estimation result includes a second movement parameter of a pixel point in the first image when moving from the first image to the second image.
[0098] In an embodiment of the present application, after generating the first information through the first machine learning model, the training device may further generate a second optical flow estimation result based on the first information. For example, the second optical flow estimation result may include a plurality of second movement parameters corresponding one-to-one to a plurality of pixels in the first image, each second movement parameter may be represented as a second vector, and each second movement parameter may be represented as a second vector. The meaning of "second vector" is similar to that of "first vector", and can be found in the description of step 403 above, which is not repeated here.
[0099] The training device can perform step 404 in a variety of ways. In one implementation, the training device can generate a second optical flow estimation result based on the first information and the position information. Alternatively, the training device can generate a second optical flow estimation result based on the first information, the position information, and the focal length of the first camera.
[0100] In which, the aforementioned position information indicates the relative position between the first position and the second position; exemplarily, the first information indicates the depth of each pixel point in the first image, and the second optical flow estimation result may include a second movement parameter of each pixel point in the first image from the first image to the second image.
[0101] Exemplarily, the above-mentioned position information may include the distance between the first position and the second position in the x-axis direction of the camera coordinate system and the distance between the first position and the second position in the y-axis direction of the camera coordinate system; or, the above-mentioned position information may also include the distance value between the first position and the second position, etc.; or, the above-mentioned position information may also include other information, which can be determined in combination with the actual application scenario and is not limited in the embodiments of the present application.
[0102] To further understand this solution, let the coordinates of any pixel point p_t in the first image be 〖(u〗_t, v_t). Two examples of formulas for calculating the second movement parameter corresponding to the pixel point p_t are disclosed below. First, refer to formula (1). The following formula (1) takes the relative position relationship between the first position and the second position as an example without limiting:
[0103] Among them, F t→s (u t ,v t ) represents pixel p t The second movement parameter from the first image to the second image, f x represents the focal length of the first camera, b x Represents the first distance when the distance between the first position and the second position is projected onto the x-axis direction of the camera coordinate system, that is, b x Represents the distance from the first position to the second position in the x-axis direction of the camera coordinate system, d t (u t ,v t ) is a value in the first information, d t (u t ,v t ) represents the coordinate (u t ,v t ) pixel p t The depth, Represents pixel p t The amount of movement in the u direction from the first image to the second image; b y Represents the second distance when the distance between the first position and the second position is projected onto the y-axis direction of the camera coordinate system, that is, b y Represents the distance from the first position to the second position in the y-axis direction of the camera coordinate system, Represents pixel p t The amount of movement in the v direction from the first image to the second image; the training device can use formula (1) to calculate the second movement parameter of each pixel in the first image. It should be noted that the example in formula (1) is only to prove the feasibility of this solution and is not used to limit this solution.
[0104] Optionally, bx and b y It is directional. If the movement from the first position to the second position is along the positive direction of the x-axis of the camera coordinate system, then b x is a positive value, will also be positive; if the movement from the first position to the second position is along the opposite direction of the x-axis of the camera coordinate system, then b x is a negative value, Correspondingly, if the movement from the first position to the second position is along the positive direction of the y-axis of the camera coordinate system, then b y is a positive value, will also be a positive value; if the movement from the first position to the second position is along the opposite direction of the y-axis of the camera coordinate system, then b y is a negative value, It can also be negative.
[0105] To understand the present solution more intuitively, please refer to Figure 5, which is a schematic diagram of the "x-axis and y-axis in the camera coordinate system" and the "u-axis and v-axis in the image coordinate system" provided in an embodiment of the present application. As shown in Figure 5, the coordinate system with the origin O1 is the camera coordinate system, and the camera coordinate system includes the x-axis and the y-axis; after projecting the camera coordinate system onto the first image, the camera coordinate system with the origin O2 is obtained. Due to the "flip" in the aforementioned projection process, the directions of the x-axis and the y-axis in the camera coordinate system with the origin O2 are shown in the figure; the directions of the u-axis and the v-axis of the image coordinate system corresponding to the first image are shown in the figure; it can be seen from Figure 5 that after projecting the camera coordinate system onto the first image, the direction of the u-axis in the image coordinate system is consistent with the direction of the x-axis in the camera coordinate system, and the direction of the v-axis in the image coordinate system is consistent with the direction of the y-axis in the camera coordinate system. It should be understood that the example in Figure 5 is only for the convenience of understanding the present solution and is not used to limit the present solution.
[0106] Refer to formula (2) again. The following formula (2) takes the first camera and the second camera as binocular cameras as an example:
[0107] Among them, since the first camera and the second camera are binocular cameras, the distance between the first position (the position when the first camera captures the first image) and the second position (the position when the second camera captures the second image) in the y-axis direction of the camera coordinate system is 0, and the movement of the pixel point in the first image in the v direction from the first image to the second image is 0. b represents the distance between the first position and the second position. Since the distance between the first position and the second position in the y-axis direction of the camera coordinate system is 0, b can also be understood as the distance between the first position and the second position in the x-axis direction of the camera coordinate system. f x represents the focal length of the first camera, d t(u t ,v t ) is a value in the first information, d t (u t ,v t ) represents the coordinate (u t ,v t ) pixel p t If the movement from the first position to the second position is in the opposite direction of the x-axis of the camera coordinate system, then Represents pixel p t The amount of movement in the u direction from the first image to the second image; if the movement from the first position to the second position is along the positive direction of the x-axis of the camera coordinate system, then use Represents pixel p t The amount of movement in the u direction from the first image to the second image. When the first camera and the second camera are binocular cameras, the training device can use formula (2) to calculate the second movement parameter of each pixel in the first image. It should be noted that the example in formula (2) is only to demonstrate the feasibility of this solution and is not used to limit this solution.
[0108] To understand this solution more intuitively, please refer to Figure 6, which is a flow chart of generating a second optical flow estimation result provided in an embodiment of the present application. As shown in Figure 6, after the first image is input into the first machine learning model, the first information output by the first machine learning model is obtained, and the first information indicates the depth of multiple pixels in the first image; based on the first information and position information (that is, the relative position between the shooting position of the first image and the shooting position of the second image), the second optical flow estimation result is generated. It should be understood that the example in Figure 6 is only for the convenience of understanding this solution and is not used to limit this solution.
[0109] In another implementation, a third machine learning model may be pre-deployed on the training device, and the training device may input the first information and the above-mentioned position information into the third machine learning model to obtain the second optical flow estimation result output by the third machine learning model; optionally, the input of the third machine learning model also includes the focal length of the first camera.
[0110] In an embodiment of the present application, since the first information indicates the depth of multiple pixels in the first image, after understanding the relative position between the first position (that is, the shooting position of the first image) and the second position (that is, the shooting position of the second image), it is possible to infer the movement parameters of the pixel points in the first image from the first image to the second image, thereby providing a feasible solution for calculating the second optical flow estimation result based on the first information; optionally, if the aforementioned reasoning process does not use a machine learning model, it is beneficial to improve the certainty of the aforementioned reasoning process, so as to reduce the possibility of introducing errors in the aforementioned reasoning process, that is, to reduce the possibility of introducing errors in the training process of the first machine learning model, which is beneficial to improve the accuracy of the information output by the trained first machine learning model.
[0111] 405. Determine a function value of a first loss function based on the first optical flow estimation result and the second optical flow estimation result, and iteratively train the first machine learning model based on the function value of the first loss function.
[0112] In an embodiment of the present application, the training device can determine the function value of the first loss function based on the first optical flow estimation result and the second optical flow estimation result, perform gradient differentiation on the function value of the first loss function, keep the parameters of the second machine learning model unchanged, and use the backpropagation algorithm to update the parameter values of the first machine learning model to complete a training of the first machine learning model. Optionally, if the second optical flow estimation result is generated by a third machine learning model, the parameters of the second machine learning model can be kept unchanged, and the backpropagation algorithm can be used to update the parameter values of the first machine learning model and the third machine learning model to complete a training of the first machine learning model.
[0113] The training device repeatedly executes steps 401 to 405 to implement iterative training of the first machine learning model (optionally, also including the third machine learning model) until the convergence condition is met; the aforementioned convergence condition may be the convergence condition of satisfying the first loss function, or it may be that the first machine learning model (optionally, also including the third machine learning model) is trained for a preset number of times.
[0114] The training device can perform step 405 in various ways. In one embodiment, the first loss function indicates the similarity between the first movement parameter and the second movement parameter corresponding to each first pixel in the first image, and the purpose of training the first machine learning model (optionally, also including the third machine learning model) using the first loss function includes improving the similarity between the first movement parameter and the second movement parameter corresponding to each first pixel; the multiple first pixels in the first image are selected from the multiple pixels included in the first image.
[0115] Before executing step 405, the training device may filter out one or more first pixel points from the multiple pixel points included in the first image; step 405 may include: the training device generates a function value of the first loss function based on the first optical flow estimation result, the second optical flow estimation result and the filtered out one or more first pixel points.
[0116] Exemplarily, the training device filters out one or more first pixel points from multiple pixel points included in the first image, which may include: the training device generates a mask corresponding to the first image; in the mask corresponding to the first image, the value of each first pixel point is 1, and the values of all other pixel points in the first image other than the first pixel are 0.
[0117] In one implementation, if step 403 is executed, the training device may further select one or more first pixel points from the multiple pixel points included in the first image based on the first optical flow estimation result and the third optical flow estimation result before executing step 405, and the confidence of the first movement parameter of each first pixel point satisfies the first condition. In the embodiment of the present application, since the first optical flow estimation result includes the first movement parameter of the pixel point in the first image when it moves from the first image to the second image, and the third optical flow estimation result includes the third movement parameter of the pixel point in the second image when it moves from the second image to the first image, ideally, the first movement parameter of the target pixel point (i.e., any pixel point) in the first image and the third movement parameter of a pixel point corresponding to the target pixel point in the second image should have the same movement value and the opposite movement direction; the greater the difference between the third movement parameter of a pixel point corresponding to the target pixel point in the second image and the movement value of the first movement parameter of the target pixel point in the first image, the lower the accuracy of the first movement parameter representing the target pixel point (i.e., the lower the confidence level); conversely, the smaller the difference between the third movement parameter of a pixel point corresponding to the target pixel point in the second image and the movement value of the first movement parameter of the target pixel point in the first image, the higher the accuracy of the first movement parameter representing the target pixel point (i.e., the higher the confidence level); therefore, it is feasible to screen multiple first pixels from the first image using the first optical flow estimation result and the third optical flow estimation result.
[0118] In addition, the loss function indicates the similarity between the first movement parameter and the second movement parameter of the first pixel point, that is, the first movement parameters of multiple first pixel points in the first optical flow estimation result are regarded as the output of the teacher model, and the first movement parameters of multiple first pixel points are used as supervision items in the training process; and the first pixel points are screened out from the multiple pixel points included in the first image, and the confidence of the first movement parameter of each first pixel point meets the first condition, that is, the accuracy of the first movement parameter of each first pixel point is higher, then the accuracy of the supervision item used in the training process is higher, which is conducive to improving the accuracy of the information output by the first machine learning model after training.
[0119] To further understand the present solution, an example of a formula used for “selecting a first pixel from a plurality of pixels included in a first image” is disclosed below:
[0120] Wherein, V1 represents a set of multiple first pixel points included in the first image, p i Represents any first pixel in V1, Represents pixel p i The first movement parameter from the first image to the second image, for example, the pixel point p in the first image i The corresponding first movement parameter can be expressed as pixel p i The corresponding first vector; Represents the pixel p in the second image determined based on the first optical flow estimation result i The coordinates of a corresponding pixel point; Represents the pixel in the second image The third moving parameter from the second image to the first image is, for example, the pixel point The corresponding third movement parameter can be expressed as a pixel point The corresponding third vector; Represents the pixel p in the first image i The corresponding first vector and pixel point in the second image The sum of the corresponding third vectors, Represents the modulus of the sum of the two vectors mentioned above, δ1 is a hyperparameter, and δ1 is a value close to 0. For example, the value of δ1 can be 0.3, 0.4 or other values, etc., which are not exhaustive here.
[0121] Since the pixel point p in the first image i and the pixels in the second image Correspondingly, under ideal conditions, that is, when both the first optical flow estimation result and the third optical flow estimation result are accurate, The value of should be 0. The smaller the value, the smaller the pixel p i The higher the confidence level of the corresponding first movement parameter, the more likely each first pixel point screened by formula (3) satisfies the constraint conditions shown in formula (3). It should be understood that formula (3) is only used to prove the feasibility of this solution and is not used to limit this solution.
[0122] In another implementation, before executing step 405, the training device may further select one or more first pixels from the plurality of pixels included in the first image, where the depth of each first pixel is less than or equal to a depth threshold. For example, the depth threshold may be 40, 50, 60, or other values, and the unit of the depth threshold may be meters. The specific design of the depth threshold can be flexibly determined based on actual application scenarios and is not limited in the embodiments of the present application.
[0123] Specifically, in one case, the training device can filter out a first pixel point whose depth is less than or equal to a depth threshold from multiple pixel points included in the first image based on the first optical flow estimation result, the focal length and position information of the first camera; the aforementioned position information indicates the relative position between the first position and the second position, the first position is the shooting position of the first image, and the second position is the shooting position of the second image.
[0124] For example, since the amount of movement of any pixel point in the first image (hereinafter referred to as the "target pixel point") in the u direction and the v direction of the image coordinate system from the first image to the second image can be calculated based on: the focal length of the first camera, the relative position between the first position and the second position, and the depth of the target pixel point, the calculation formula can refer to the description in the above formula (1) and formula (2), which will not be repeated here. Referring to the above formula, it can be seen that, excluding the case where the distance value between the first position and the second position in the x-axis direction or the y-axis direction of the image coordinate system is 0, under the premise that the relative position between the first position and the second position and the focal length of the first camera are fixed, the greater the depth of the target pixel point, the smaller the movement distance of the target pixel point in the u direction and the v direction from the first image to the second image should be. Correspondingly, if the movement distance of the target pixel point in the u direction and the v direction from the first image to the second image is smaller, it represents that the depth of the target pixel point is larger, thereby achieving "filtering out the pixel points whose depth is less than or equal to the depth threshold from the multiple pixels included in the first image based on the first optical flow estimation result, the focal length and position information of the first camera".
[0125] To further understand this solution, two examples of formulas used for “selecting a first pixel from a plurality of pixels included in a first image” are disclosed below. First, refer to formula (4). The following formula (4) takes the relative position relationship between the first position and the second position as an example without limiting the relative position relationship between the first position and the second position:
[0126] Wherein, V2 represents a set of multiple first pixel points included in the first image, p i Represents any first pixel in V1, Represents pixel p i The first movement parameter from the first image to the second image, for example, the pixel point p in the first image i The corresponding first movement parameter can be expressed as pixel p i The first vector corresponding to Represents pixel p i The modulus of the first vector corresponding to The calculated pixel point p i The distance value moved from the first image to the second image; δ2 represents the depth threshold, Represents when pixel p i The distance from the first image to the second image when the depth is δ2. That is, pixel p i The distance from the first image to the second image is greater than the pixel p i The distance from the first image to the second image when the depth is δ2 can be deduced from the pixel point p i The depth is less than or equal to δ2. It should be understood that formula (4) is only used to prove the feasibility of this solution and is not used to limit this solution.
[0127] Referring to formula (5) again, the following formula (5) takes the first camera and the second camera as binocular cameras as an example:
[0128] Among them, V2, as well as The concept of can refer to the description of formula (4) above. The difference is that since the first camera and the second camera are binocular cameras, the pixel point p i The movement amount in the v direction of the image coordinate system from the first image to the second image is 0, then Can represent based on The calculated pixel point p i The moving distance in the u direction of the image coordinate system from the first image to the second image; Represents when pixel p iThe moving distance in the u direction of the image coordinate system from the first image to the second image when the depth is δ2; The pixel point p can be derived i The depth is less than or equal to δ2. It should be understood that formula (5) is only used to prove the feasibility of this solution and is not used to limit this solution.
[0129] In an embodiment of the present application, since the first optical flow estimation result is generated by a first machine learning model that has performed a training operation, that is, the accuracy of the first optical flow estimation result is relatively high, using the first optical flow estimation result to perform the operation of "screening out the first pixel point in the first image whose depth is less than or equal to the depth threshold" is beneficial to improving the accuracy of the aforementioned operation.
[0130] In another case, the training device can filter out the first pixel point whose depth is less than or equal to the depth threshold from the multiple pixel points included in the first image based on the first information, where the first information indicates the depth value of each pixel point in the first image, etc. The training device can also use other methods to filter out the first pixel point whose depth is less than or equal to the depth threshold from the multiple pixel points included in the first image, which is not limited in the embodiments of the present application.
[0131] In an embodiment of the present application, since when the depth of a pixel point in the first image is too large, the error in predicting the movement parameters of the aforementioned pixel point from the first image to the second image is likely to be larger, and the depth of each first pixel point is less than or equal to the depth threshold, the accuracy of the first movement parameter of each first pixel point is higher, that is, the accuracy of the supervision item used in the training process is higher, which is beneficial to improving the accuracy of the information output by the first machine learning model after training.
[0132] In another implementation, if step 403 is executed, the training device may further determine, before executing step 405, one or more first pixels from the multiple pixels included in the first image based on the first optical flow estimation result and the third optical flow estimation result, where the confidence level of the first motion parameter of each first pixel satisfies the first condition and the depth of each first pixel is less than or equal to the depth threshold. For example, the set of all first pixels included in the first image is the intersection of V_1 and V_2. The method for determining V_1 and V_2 can be as described above and will not be repeated here.
[0133] To understand this solution more intuitively, please refer to Figure 7, which is a schematic diagram of a set of all first pixel points included in the first image provided in an embodiment of the present application. As shown in Figure 7, the training device combines the first image and the second image and inputs them into a second machine learning model that has performed a training operation. When the optical flow from the first image to the second image is generated by the second machine learning model, the first optical flow estimation result output by the second machine learning model is obtained; when the optical flow from the second image to the first image is generated by the second machine learning model, the third optical flow estimation result output by the second machine learning model is obtained. The training device can filter out a first set of pixel points from the first image based on the first optical flow estimation result and the third optical flow estimation result, and the confidence of the first movement parameter of each pixel point in the first pixel point combination meets the first condition; based on the first optical flow estimation result, filter out a second set of pixel points from the first image, and the depth of each pixel point in the second pixel point combination is less than or equal to the depth threshold; the training device determines the intersection of the first pixel point set and the second pixel point set as the set consisting of all first pixel points in the first image; Figure 7 shows the set consisting of all first pixel points in the first image in a visual form, the first pixel point in the first image is black, and all other pixel points in the first image other than the first pixel point are white. It should be understood that the specific implementation methods of the multiple steps involved in Figure 7 can be referred to the above description and will not be repeated here. The example in Figure 7 is only for the convenience of understanding this solution and is not used to limit this solution.
[0134] To further understand this solution, an example of the calculation formula of the first loss function is disclosed below:
[0135] in, represents the first loss function, Indicates any pixel point p in the first image i The similarity between the corresponding first movement parameter and the third movement parameter is calculated in formula (6) t→s (p i )and For example, the L1 distance between f (p i The specific expression of ) can be found in the description of formula (7). According to formula (7), when the pixel point p i ∈(V1∩V2), that is, when the pixel point p i The confidence of the first movement parameter satisfies the first condition and the pixel point p i When the depth of the pixel p is less than or equal to the depth threshold, the pixel p i The L1 distance between the corresponding first movement parameter and the third movement parameter is retained. iWhen it does not belong to (V1∩V2), the pixel point p i The L1 distance between the corresponding first movement parameter and the third movement parameter is discarded. It should be understood that the examples in formula (6) and formula (7) are only for facilitating the understanding of this solution and are not used to limit this solution.
[0136] For a more intuitive understanding of this solution, please refer to Figure 8, which is another flow chart of the training method of the model provided in an embodiment of the present application. Figure 8 can be understood in conjunction with the above description of Figures 6 and 7. The above description of Figures 6 and 7 has explained how to obtain the second optical flow estimation result, the first optical flow estimation result, and how to determine all first pixel points in the first image, which will not be repeated here.
[0137] After obtaining the first optical flow estimation result, the second optical flow estimation result, and determining all first pixel points in the first image, the training device can generate a first loss function (i.e., ), and updates the parameters of the first machine learning model according to the function value of the first loss function.
[0138] Among them, the first loss function indicates the pixel degree between the first movement parameter and the second movement parameter corresponding to each first pixel point, the first movement parameter corresponding to each first pixel point comes from the first optical flow estimation result, and the first movement parameter corresponding to each second pixel point comes from the second optical flow estimation result. It should be understood that the example in Figure 8 is only for the convenience of understanding this scheme and is not used to limit this scheme.
[0139] In another embodiment, the first loss function indicates the similarity between the first optical flow estimation result and the second optical flow estimation result, and the purpose of training the first machine learning model (optionally, the third machine learning model) using the first loss function includes improving the similarity between the first optical flow estimation result and the second optical flow estimation result. In step 405, the training device can directly calculate the similarity between the first optical flow estimation result and the second optical flow estimation result, thereby obtaining the function value of the first loss function.
[0140] In an embodiment of the present application, the loss function is obtained based on the first optical flow estimation result and the second optical flow estimation result. The first optical flow estimation result is generated by a second machine learning model that has performed a training operation. The accuracy of the first optical flow estimation result is relatively high. That is, the process of training the first machine learning model using the aforementioned loss function introduces the idea of knowledge distillation, and provides a solution for training the first machine learning model in a self-supervised manner. In addition, the optical flow estimation result is generated based on the depth information output by the first machine learning model. The execution process of the aforementioned steps is less affected by factors such as the different light when the first image and the second image are collected. That is, the accuracy of the aforementioned steps is relatively high, thereby reducing the additional error introduced by the step of "generating the second optical flow estimation result based on the first information", which is conducive to improving the accuracy of the results output by the trained first machine learning model.
[0141] 2. Application Phase
[0142] In the embodiment of the present application, the application stage describes a process in which the execution device 230 uses the first machine learning model 201 to generate the depth of pixels in the image to be processed. Specifically, please refer to FIG9 , which is a flow chart of an image processing method provided in the embodiment of the present application. The image processing method provided in the embodiment of the present application may include:
[0143] 901. Obtain an image to be processed.
[0144] 902. Input the image to be processed into the first machine learning model to obtain second information output by the first machine learning model, where the second information indicates the depth of the pixel points in the image to be processed; wherein the first machine learning model is trained using a loss function, and the loss function is obtained based on the first optical flow estimation result and the second optical flow estimation result, the first optical flow estimation result includes a first movement parameter of the pixel points in the first image when moving from the first image to the second image, and the second optical flow estimation result includes a second movement parameter of the pixel points in the first image when moving from the first image to the second image; the first optical flow estimation result is generated by the second machine learning model that has performed a training operation, and the second optical flow estimation result is obtained based on the first information indicating the depth of the pixel points in the first image, and when the output of the first machine learning model is the first information, the input of the first machine learning model is the first image.
[0145] In an embodiment of the present application, after determining the image to be processed, the execution device can input the image to be processed into a first machine learning model that has performed a training operation to obtain second information output by the first machine learning model. The second information indicates the depth of the pixel points in the image to be processed. The specific expression of the "second information" is similar to the specific expression of the "first information". Please refer to the description in the corresponding embodiment of Figure 4 above, and no further details will be given here.
[0146] The first machine learning model is obtained by training through the embodiment corresponding to Figure 4. The meanings of each noun in step 901 and step 902 can be found in the description in the embodiment corresponding to Figure 4 above, and will not be repeated here.
[0147] In order to further illustrate the beneficial effects of the method provided by the present application, the following is an explanation based on the experimental data in Table 1.
[0148] Table 1
[0149] Absolute relative error (AbsRel) and δ^1 are two indicators that measure the accuracy of the depth information output by the trained first machine learning model. The smaller the value of AbsRel, the greater the accuracy of the depth information output by the first machine learning model; the larger the value of δ^1, the greater the accuracy of the depth information output by the first machine learning model. The data in Table 1 show that after the first machine learning model is trained using the method provided in this application, the depth information output by the first machine learning model is more accurate.
[0150] Among them, L P Represents the use of traditional methods to train the first machine learning; L fd represents training the first machine learning model using the first loss function, where the first loss function indicates the similarity between the first optical flow estimation result and the second optical flow estimation result (i.e., the training method 1 in the method provided in this application); L fd +M f It represents training the first machine learning model using the first loss function, where the first loss function indicates the similarity between the first movement parameter and the second movement parameter corresponding to each first pixel point (that is, the second training method in the method provided in this application).
[0151] Absolute relative error (Abs Rel) and δ 1 These are two indicators for measuring the accuracy of the depth information output by the trained first machine learning model. The smaller the value of Abs Rel, the greater the accuracy of the depth information output by the first machine learning model. 1 The larger the value of , the greater the accuracy of the depth information output by the first machine learning model. It can be seen from the data in Table 1 that after the first machine learning model is trained using the method provided in this application, the accuracy of the depth information output by the first machine learning model is higher.
[0152] On the basis of the embodiments corresponding to Figures 1 to 9, in order to better implement the above-mentioned scheme of the embodiment of the present application, the following also provides related devices for implementing the above-mentioned scheme. Referring specifically to Figure 10, Figure 10 is a structural schematic diagram of a training device for a model provided in an embodiment of the present application, and the model training device 1000 includes: an input module 1001, which is used to input a first image into a first machine learning model to obtain first information output by the first machine learning model, where the first information indicates the depth of a pixel in the first image; the input module 1001 is also used to input the first image and the second image into a second machine learning model, and generate a first optical flow estimation result through the second machine learning model, wherein the second machine learning model is a machine learning model that has performed a training operation, and the first optical flow estimation result includes a first movement parameter of a pixel in the first image when it moves from the first image to the second image; a generation module 1002 is used to generate a second optical flow estimation result based on the first information, and the second optical flow estimation result includes a second movement parameter of a pixel in the first image when it moves from the first image to the second image; a training module 1003 is used to determine a function value of a loss function based on the first optical flow estimation result and the second optical flow estimation result, and train the first machine learning model based on the function value of the loss function.
[0153] Optionally, the input module 1001 is also used to input the second image and the first image into the second machine learning model, and generate a third optical flow estimation result through the second machine learning model, wherein the third optical flow estimation result includes a third movement parameter of the pixel point in the second image from the second image to the first image; wherein the loss function indicates the similarity between the first movement parameter and the second movement parameter corresponding to the first pixel point in the first image, the confidence of the first movement parameter of the first pixel point satisfies the first condition, and the confidence of the first movement parameter of the first pixel point is determined based on the first optical flow estimation result and the third optical flow estimation result.
[0154] Optionally, the loss function indicates a similarity between a first movement parameter and a second movement parameter corresponding to a first pixel in the first image, and a depth of the first pixel is less than or equal to a depth threshold.
[0155] Optionally, the model training device 1000 also includes: a determination module, used to determine the pixel points in the first image whose depth is less than or equal to the depth threshold based on the first optical flow estimation result, the focal length and position information of the first camera, wherein the position information indicates the relative position between the first position and the second position, the first position is the shooting position of the first image, and the second position is the shooting position of the second image.
[0156] Optionally, the generation module 1002 is specifically used to generate a second optical flow estimation result based on the first information and position information, the position information indicates the relative position between the first position and the second position, the first position is the shooting position of the first image, and the second position is the shooting position of the second image.
[0157] Optionally, the first image and the second image are from the same camera, or the first image is from two different cameras.
[0158] It should be noted that the information interaction, execution process, etc. between the modules / units in the model training device 1000 are based on the same concept as the various method embodiments corresponding to Figures 3 to 8 in this application. For specific contents, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.
[0159] Please continue to refer to Figure 11, which is a structural diagram of an image processing device provided by an embodiment of the present application. The image processing device 1100 may include: an acquisition module 1101, used to acquire an image to be processed; an input module 1102, used to input the image to be processed into a first machine learning model to obtain second information output by the first machine learning model, and the second information indicates the depth of the pixel points in the image to be processed; wherein the first machine learning model is trained using a loss function, and the loss function is obtained based on the first optical flow estimation result and the second optical flow estimation result, the first optical flow estimation result includes a first movement parameter of the pixel points in the first image from the first image to the second image, and the second optical flow estimation result includes a second movement parameter of the pixel points in the first image from the first image to the second image; the first optical flow estimation result is generated by a second machine learning model that has performed a training operation, and the second optical flow estimation result is obtained based on the first information indicating the depth of the pixel points in the first image. When the output of the first machine learning model is the first information, the input of the first machine learning model is the first image.
[0160] Optionally, the loss function indicates the similarity between a first movement parameter and a second movement parameter corresponding to a first pixel point in the first image, and the confidence of the first movement parameter of the first pixel point satisfies a first condition; the confidence of the first movement parameter of the first pixel point is determined based on the first optical flow estimation result and a third optical flow estimation result, the third optical flow estimation result is generated by a second machine learning model, and the third optical flow estimation result includes a third movement parameter of the pixel point in the second image from the second image to the first image.
[0161] Optionally, the loss function indicates a similarity between a first movement parameter and a second movement parameter corresponding to a first pixel in the first image, and a depth of the first pixel is less than or equal to a depth threshold.
[0162] It should be noted that the information interaction, execution process, etc. between the modules / units in the image processing device 1100 are based on the same concept as the various method embodiments corresponding to Figure 9 in this application. For specific contents, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.
[0163] Next, an execution device provided in an embodiment of the present application will be described. Please refer to FIG12 , which is a schematic structural diagram of an execution device provided in an embodiment of the present application. Specifically, the execution device 1200 includes: a receiver 1201, a transmitter 1202, a processor 1203, and a memory 1204 (wherein the number of processors 1203 in the execution device 1200 may be one or more, and FIG12 uses one processor as an example). The processor 1203 may include an application processor 12031 and a communication processor 12032. In some embodiments of the present application, the receiver 1201, the transmitter 1202, the processor 1203, and the memory 1204 may be connected via a bus or other means.
[0164] The memory 1204 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1203. A portion of the memory 1204 may also include non-volatile random access memory (NVRAM). The memory 1204 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.
[0165] Processor 1203 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.
[0166] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1203. Processor 1203 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1203. The above processor 1203 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1203 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1204, and processor 1203 reads the information in memory 1204 and, in conjunction with its hardware, completes the steps of the above method.
[0167] Receiver 1201 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1202 can be used to output digital or character information through the first interface. Transmitter 1202 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1202 can also include a display device such as a display screen.
[0168] In the embodiment of the present application, the processor 1203 is configured to execute the image processing method executed by the execution device in the embodiment corresponding to FIG9 . It should be noted that the specific manner in which the application processor 12031 in the processor 1203 executes the aforementioned steps is based on the same concept as the various method embodiments corresponding to FIG9 in the present application, and the technical effects brought about are the same as the various method embodiments corresponding to FIG9 in the present application. For specific details, please refer to the description of the method embodiments shown above in the present application, and will not be repeated here.
[0169] The present application also provides a training device. Please refer to FIG. 13 , which is a schematic diagram of the structure of a training device provided by an embodiment of the present application. Specifically, the training device 1300 is implemented by one or more servers. The training device 1300 may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1322 (e.g., one or more processors) and memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) storing application programs 1342 or data 1344. The memory 1332 and storage medium 1330 may be either ephemeral or persistent storage. The program stored in the storage medium 1330 may include one or more modules (not shown), each of which may include a series of instruction operations on the training device. Furthermore, the CPU 1322 may be configured to communicate with the storage medium 1330 to execute the series of instruction operations in the storage medium 1330 on the training device 1300.
[0170] The training device 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1358, and / or one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0171] In the embodiment of the present application, the central processing unit 1322 is used to execute the training method of the model executed by the training device in the embodiments corresponding to Figures 3 to 8. It should be noted that the specific manner in which the central processing unit 1322 executes the above-mentioned steps is based on the same concept as the various method embodiments corresponding to Figures 3 to 8 in this application, and the technical effects brought about are the same as the various method embodiments corresponding to Figures 3 to 8 in this application. For specific details, please refer to the description of the method embodiments shown above in this application, and will not be repeated here.
[0172] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program for signal processing. When the computer-readable storage medium is run on a computer, the computer executes the steps executed by the training device in the method described in the embodiments shown in Figures 3 to 8 above, or the computer executes the steps executed by the execution device in the method described in the embodiment shown in Figure 9 above.
[0173] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps performed by the training device in the method described in the embodiments shown in Figures 3 to 8 above, or enables the computer to execute the steps performed by the execution device in the method described in the embodiment shown in Figure 9 above.
[0174] The execution device and training device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip executes the training method of the model described in the embodiments shown in Figures 3 to 8 above, or so that the chip executes the image processing method described in the embodiment shown in Figure 9 above. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
[0175] Specifically, see Figure 14 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be implemented as a neural network processor (NPU) 140. NPU 140 is mounted on a host CPU (host CPU) as a coprocessor, with tasks assigned by the host CPU. The core of the NPU is arithmetic circuit 1403, which is controlled by controller 1404 to extract matrix data from memory and perform multiplication operations.
[0176] In some implementations, the arithmetic circuit 1403 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1403 is a two-dimensional systolic array. The arithmetic circuit 1403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1403 is a general-purpose matrix processor.
[0177] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1408.
[0178] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 through the Direct Memory Access Controller (DMAC) 1405. Input data is also transferred to unified memory 1406 through the DMAC.
[0179] BIU stands for Bus Interface Unit, i.e., bus interface unit 1410 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1409 .
[0180] The bus interface unit 1410 (BIU) is used for the instruction fetch memory 1409 to obtain instructions from the external memory, and is also used for the storage unit access controller 1405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
[0181] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1406 or transfer weight data to the weight memory 1402 or transfer input data to the input memory 1401.
[0182] The vector calculation unit 1407 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.
[0183] In some implementations, the vector calculation unit 1407 can store the processed output vector to the unified memory 1406. For example, the vector calculation unit 1407 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1403, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values to generate an activation value. In some implementations, the vector calculation unit 1407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1403, for example, for use in a subsequent layer in a neural network.
[0184] An instruction fetch buffer 1409 connected to the controller 1404 is used to store instructions used by the controller 1404;
[0185] Unified memory 1406, input memory 1401, weight memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memories are private to the NPU hardware architecture.
[0186] Among them, the operations of each layer in the machine learning model shown in Figures 3 to 9 can be performed by the operation circuit 1403 or the vector calculation unit 1407.
[0187] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.
[0188] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0189] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.
[0190] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0191] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, training device or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive (SSD)).
Claims
1. A model training method, characterized in that: The method comprises: Inputting a first image into a first machine learning model to obtain first information output by the first machine learning model, wherein the first information indicates the depth of a pixel in the first image; Inputting the first image and the second image into a second machine learning model, and generating a first optical flow estimation result through the second machine learning model, wherein the second machine learning model is a machine learning model that has performed a training operation, and the first optical flow estimation result includes a first movement parameter of a pixel point in the first image from the first image to the second image; generating a second optical flow estimation result according to the first information, wherein the second optical flow estimation result includes a second movement parameter of a pixel point in the first image from the first image to the second image; Determine a function value of a loss function according to the first optical flow estimation result and the second optical flow estimation result, and train the first machine learning model according to the function value of the loss function.
2. The method according to claim 1, characterized in that The method further comprises: Inputting the second image and the first image into the second machine learning model, generating a third optical flow estimation result through the second machine learning model, wherein the third optical flow estimation result includes a third movement parameter of a pixel point in the second image from the second image to the first image; Wherein, the loss function indicates the similarity between the first movement parameter and the second movement parameter corresponding to the first pixel point in the first image, the confidence of the first movement parameter of the first pixel point satisfies the first condition, and the confidence of the first movement parameter of the first pixel point is determined based on the first optical flow estimation result and the third optical flow estimation result.
3. The method according to claim 1 or 2, characterized in that: The loss function indicates a similarity between the first movement parameter and the second movement parameter corresponding to a first pixel in the first image, wherein a depth of the first pixel is less than or equal to a depth threshold.
4. The method according to claim 3, characterized in that: The method further comprises: Determine, according to the first optical flow estimation result, the focal length and position information of the first camera, pixel points in the first image whose depth is less than or equal to the depth threshold, wherein the position information indicates a relative position between a first position and a second position, the first position is a shooting position of the first image, and the second position is a shooting position of the second image.
5. The method according to claim 1 or 2, characterized in that: Generating a second optical flow estimation result according to the first information includes: The second optical flow estimation result is generated according to the first information and position information, wherein the position information indicates a relative position between a first position and a second position, the first position is a shooting position of the first image, and the second position is a shooting position of the second image.
6. The method according to claim 1 or 2, characterized in that: The first image and the second image are from the same camera, or the first image is from two different cameras.
7. An image processing method, characterized in that: The method comprises: Get the image to be processed; Inputting the image to be processed into a first machine learning model to obtain second information output by the first machine learning model, wherein the second information indicates the depth of a pixel in the image to be processed; The first machine learning model is trained by using a loss function, the loss function is obtained according to a first optical flow estimation result and a second optical flow estimation result, the first optical flow estimation result includes a first movement parameter of a pixel point in the first image from the first image to the second image, and the second optical flow estimation result includes a second movement parameter of a pixel point in the first image from the first image to the second image; The first optical flow estimation result is generated by a second machine learning model that has performed a training operation, and the second optical flow estimation result is obtained based on first information indicating the depth of a pixel point in the first image. When the output of the first machine learning model is the first information, the input of the first machine learning model is the first image.
8. The method according to claim 7, characterized in that The loss function indicates a similarity between the first movement parameter and the second movement parameter corresponding to a first pixel in the first image, and a confidence level of the first movement parameter of the first pixel satisfies a first condition; The confidence of the first movement parameter of the first pixel point is determined based on the first optical flow estimation result and a third optical flow estimation result, the third optical flow estimation result is generated by the second machine learning model, and the third optical flow estimation result includes the third movement parameter of the pixel point in the second image from the second image to the first image.
9. The method according to claim 7 or 8, characterized in that: The loss function indicates a similarity between the first movement parameter and the second movement parameter corresponding to a first pixel in the first image, wherein a depth of the first pixel is less than or equal to a depth threshold.
10. A model training device, characterized in that: The device comprises: An input module, configured to input a first image into a first machine learning model to obtain first information output by the first machine learning model, wherein the first information indicates the depth of a pixel in the first image; The input module is further used to input the first image and the second image into a second machine learning model, and generate a first optical flow estimation result through the second machine learning model, wherein the second machine learning model is a machine learning model that has performed a training operation, and the first optical flow estimation result includes a first movement parameter of a pixel point in the first image from the first image to the second image; a generating module, configured to generate a second optical flow estimation result according to the first information, wherein the second optical flow estimation result includes a second movement parameter of a pixel point in the first image from the first image to the second image; A training module is used to determine a function value of a loss function according to the first optical flow estimation result and the second optical flow estimation result, and to train the first machine learning model according to the function value of the loss function.
11. The device according to claim 10, characterized in that The input module is further used to input the second image and the first image into the second machine learning model, and generate a third optical flow estimation result through the second machine learning model, wherein the third optical flow estimation result includes a third movement parameter of a pixel point in the second image from the second image to the first image; Wherein, the loss function indicates the similarity between the first movement parameter and the second movement parameter corresponding to the first pixel point in the first image, the confidence of the first movement parameter of the first pixel point satisfies the first condition, and the confidence of the first movement parameter of the first pixel point is determined based on the first optical flow estimation result and the third optical flow estimation result.
12. The device according to claim 10 or 11, characterized in that The loss function indicates a similarity between the first movement parameter and the second movement parameter corresponding to a first pixel in the first image, wherein a depth of the first pixel is less than or equal to a depth threshold.
13. The device according to claim 12, characterized in that The device also includes: A determination module is used to determine, according to the first optical flow estimation result, the focal length and position information of the first camera, the pixel points in the first image whose depth is less than or equal to the depth threshold, wherein the position information indicates the relative position between the first position and the second position, the first position is the shooting position of the first image, and the second position is the shooting position of the second image.
14. The device according to claim 10 or 11, characterized in that The generation module is specifically used to generate the second optical flow estimation result according to the first information and position information, the position information indicates the relative position between the first position and the second position, the first position is the shooting position of the first image, and the second position is the shooting position of the second image.
15. The device according to claim 10 or 11, characterized in that The first image and the second image are from the same camera, or the first image is from two different cameras.
16. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed; An input module, used for inputting the image to be processed into a first machine learning model to obtain second information output by the first machine learning model, wherein the second information indicates the depth of a pixel in the image to be processed; The first machine learning model is trained by using a loss function, the loss function is obtained according to a first optical flow estimation result and a second optical flow estimation result, the first optical flow estimation result includes a first movement parameter of a pixel point in the first image from the first image to the second image, and the second optical flow estimation result includes a second movement parameter of a pixel point in the first image from the first image to the second image; The first optical flow estimation result is generated by a second machine learning model that has performed a training operation, and the second optical flow estimation result is generated by a second machine learning model that has performed a training operation. The first information indicating the depth of the pixel point in the first image is obtained. When the output of the first machine learning model is the first information, the input of the first machine learning model is the first image.
17. The device according to claim 16, characterized in that The loss function indicates a similarity between the first movement parameter and the second movement parameter corresponding to a first pixel in the first image, and a confidence level of the first movement parameter of the first pixel satisfies a first condition; The confidence of the first movement parameter of the first pixel point is determined based on the first optical flow estimation result and a third optical flow estimation result, the third optical flow estimation result is generated by the second machine learning model, and the third optical flow estimation result includes the third movement parameter of the pixel point in the second image from the second image to the first image.
18. The device according to claim 16 or 17, characterized in that The loss function indicates a similarity between the first movement parameter and the second movement parameter corresponding to a first pixel in the first image, wherein a depth of the first pixel is less than or equal to a depth threshold.
19. An execution device, characterized in that: comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is used to execute the program in the memory, so that the execution device executes the method according to any one of claims 7 to 9.
20. A training device, characterized in that comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is used to execute the program in the memory so that the training device performs the method as claimed in any one of claims 1 to 6.
21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.
22. A computer program product, characterized in that The computer program product comprises a program, and when the program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Depth and confidence estimation system
CN112488104A
Model training method and device, video frame insertion method and device, equipment and storage medium
CN113365110A
Method for eliminating uncertainty of self-supervised three-dimensional reconstruction
CN113592913A
Unsupervised training of optical flow estimation neural networks
CN115222777A
Unsupervised Video Segmentation
US20190188863A1
