Optical flow determination method, model training method and device for determining optical flow

Abstract

The method comprises the following steps: determining a true value of a first optical flow based on point cloud data sets and images corresponding to a target road; determining a predicted value of the first optical flow based on the true value of the first optical flow; determining a predicted value of a second optical flow based on the predicted value of the first optical flow and a corresponding relationship between the first optical flow and the second optical flow; the first optical flow is a forward optical flow or a reverse optical flow; the forward optical flow is an optical flow from a perspective of an imaging device for shooting images to a virtual perspective; and the reverse optical flow is an optical flow from the virtual perspective to the perspective of the imaging device. The predicted value of the first optical flow is denser than the true value of the first optical flow, that is, a denser predicted value of the first optical flow can be determined by the method. Moreover, the predicted value of the first optical flow is denser, and the predicted value of the second optical flow determined is also denser. Therefore, the method can obtain dense first and second optical flows.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of assisted driving, and particularly to a method for determining optical flow, a model training method and device for determining optical flow. BACKGROUND

[0002] In application scenarios such as assisted driving or automatic driving, optical flow data is sometimes needed. The commonly used method is to first obtain the correspondence between the point cloud data set obtained by the vehicle-mounted laser radar and the image captured by the imaging device, and then obtain the optical flow data based on the correspondence.

[0003] However, the point cloud data set obtained by the above method is usually sparse, which leads to the optical flow data obtained by the prior art being also sparse, and the sparse optical flow data is difficult to meet the needs of application scenarios such as assisted driving or automatic driving. SUMMARY

[0004] To solve the above technical problems, the present disclosure is proposed. Embodiments of the present disclosure provide a method for determining optical flow, a model training method and device for determining optical flow.

[0005] According to one aspect of the present disclosure, a method for determining optical flow is provided, comprising:

[0006] determining a true value of a first optical flow based on a point cloud data set and an image corresponding to a target road respectively;

[0007] determining a predicted value of the first optical flow based on the true value of the first optical flow, the predicted value of the first optical flow being denser than the true value of the first optical flow;

[0008] determining a predicted value of a second optical flow based on the predicted value of the first optical flow and a correspondence between the first optical flow and the second optical flow;

[0009] wherein the first optical flow is a forward optical flow or a reverse optical flow, the forward optical flow is an optical flow from a perspective of an imaging device capturing the image to a virtual perspective, and the reverse optical flow is an optical flow from the virtual perspective to the perspective of the imaging device.

[0010] According to yet another aspect of the embodiments of the present disclosure, a model training method for determining optical flow is provided, comprising:

[0011] determining a true value of a first optical flow based on a method for determining optical flow according to the above aspect, the true value of the first optical flow being determined based on a point cloud data set and an image corresponding to a target road respectively;

[0012] processing the point cloud data set and the image by using a neural network model to be trained to obtain a predicted value of the first optical flow;

[0013] determine a network loss based on the ground truth of the first optical flow and the predicted value of the first optical flow;

[0014] train the neural network model based on the network loss;

[0015] The first optical flow is a forward optical flow or a reverse optical flow. The forward optical flow is an optical flow from a perspective of an imaging device that captures the image to a virtual perspective. The reverse optical flow is an optical flow from the virtual perspective to the perspective of the imaging device.

[0016] According to yet another aspect of the embodiments of the present disclosure, a device for determining an optical flow is provided, which comprises:

[0017] A first ground truth determination module is configured to determine a ground truth of a first optical flow based on point cloud data sets and images corresponding to a target road respectively.

[0018] A first predicted value determination module is configured to determine a predicted value of the first optical flow based on the ground truth of the first optical flow determined by the first ground truth determination module, the predicted value of the first optical flow being denser than the ground truth of the first optical flow.

[0019] A second predicted value determination module is configured to determine a predicted value of a second optical flow based on the predicted value of the first optical flow determined by the first predicted value determination module and a corresponding relationship between the first optical flow and the second optical flow.

[0020] The first optical flow is a forward optical flow or a reverse optical flow. The forward optical flow is an optical flow from a perspective of an imaging device that captures the image to a virtual perspective. The reverse optical flow is an optical flow from the virtual perspective to the perspective of the imaging device.

[0021] According to yet another aspect of the embodiments of the present disclosure, a model training device for determining an optical flow is provided, which comprises:

[0022] An optical flow ground truth determination module is configured to determine a ground truth of a first optical flow based on an optical flow determination method according to any one of claims 1-6, the ground truth of the first optical flow being determined based on point cloud data sets and images corresponding to a target road respectively.

[0023] An optical flow predicted value determination module is configured to process the point cloud data sets and the images by using a neural network model to be trained to obtain a predicted value of the first optical flow.

[0024] A network loss determination module is configured to determine a network loss based on the ground truth of the first optical flow determined by the optical flow ground truth determination module and the predicted value of the first optical flow determined by the optical flow predicted value determination module.

[0025] a network model training module, configured to train the neural network model based on the network loss determined by the network loss determination module;

[0026] The first optical flow is a forward optical flow or a reverse optical flow. The forward optical flow is an optical flow from a perspective of an imaging device that captures the image to the virtual perspective. The reverse optical flow is an optical flow from the virtual perspective to the perspective of the imaging device.

[0027] According to a further aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program for performing the optical flow determination method according to any one of the embodiments of the present disclosure or the model training method for determining an optical flow.

[0028] According to a further aspect of the embodiments of the present disclosure, an electronic device is provided, which comprises:

[0029] a processor;

[0030] a memory for storing the executable instructions of the processor;

[0031] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the optical flow determination method according to any one of the embodiments of the present disclosure or the model training method for determining an optical flow.

[0032] By the method provided in the embodiments of the present disclosure, after the true value of the first optical flow is determined, the predicted value of the first optical flow is determined based on the true value of the first optical flow, and the predicted value of the second optical flow is determined based on the predicted value of the first optical flow and the corresponding relationship between the first optical flow and the second optical flow. The predicted value of the first optical flow is denser than the true value of the first optical flow. Therefore, by the method provided in the embodiments of the present disclosure, the predicted value of the first optical flow can be determined to be denser. Further, since the predicted value of the first optical flow is denser, the predicted value of the second optical flow is also denser. That is, by the method provided in the embodiments of the present disclosure, the first optical flow and the second optical flow can be obtained to be denser, and the problem that only sparse optical flow can be obtained in the prior art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0033] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0034] Figure 1 is a schematic diagram of a scenario to which the present disclosure is applicable.

[0035] Figure 2 FIG. 1 is a flowchart of a method for determining an optical flow according to an example embodiment of the present disclosure.

[0036] Figure 3 FIG. 2 is a flowchart of a method for determining an optical flow according to another example embodiment of the present disclosure.

[0037] Figure 4 FIG. 3 is a flowchart of a method for determining an optical flow according to another example embodiment of the present disclosure.

[0038] Figure 5 FIG. 4 is a flowchart of a method for determining an optical flow according to another example embodiment of the present disclosure.

[0039] Figure 6 FIG. 5 is a flowchart of a method for training a model for determining an optical flow according to an example embodiment of the present disclosure.

[0040] Figure 7 FIG. 6 is a structural diagram of an apparatus for determining an optical flow according to an example embodiment of the present disclosure.

[0041] Figure 8 FIG. 7 is a structural diagram of an apparatus for determining an optical flow according to another example embodiment of the present disclosure.

[0042] Figure 9 FIG. 8 is a structural diagram of an apparatus for training a model for determining an optical flow according to an example embodiment of the present disclosure.

[0043] Figure 10 FIG. 9 is a structural diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] Hereinafter, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, and the present disclosure is not limited to the described example embodiments.

[0045] It should be noted that the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0046] Those skilled in the art can understand that the terms "first", "second", and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they represent a necessary logical sequence between them.

[0047] It should also be understood that in the embodiments of the present disclosure, "a plurality of" can mean two or more, and "at least one" can mean one, two, or more.

[0048] It should also be understood that, with respect to any of the components, data, or structures mentioned in the embodiments of the present disclosure, one or more can be generally understood unless specifically limited or unless the contrary is indicated by the context.

[0049] In addition, the term "and / or" in the present disclosure is merely a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0050] It should also be understood that the description of the embodiments of the present disclosure emphasizes the differences between the embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0051] At the same time, it should be understood that, for the convenience of description, the size of each part shown in the drawings is not drawn according to the actual proportion relationship.

[0052] The following description of at least one example embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0053] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the specification where appropriate.

[0054] It should be noted that similar reference numbers and letters refer to similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0055] The embodiments of the present disclosure can be applied to terminal devices, computer systems, servers, and other electronic devices, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, and the like.

[0056] Electronic devices such as terminal devices, computer systems, servers, and the like can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. Computer systems / server can be practiced in distributed cloud computing environments with other computer systems coupled via communication networks. The program modules can be stored in the local or remote computer system memory devices.

[0057] SUMMARY

[0058] In application scenarios such as assisted driving or automatic driving, sometimes it is necessary to apply optical flow data. The optical flow can be the optical flow formed by images of the same camera at different times, or can be the optical flow formed according to the conversion of the viewing angles of different sensors.

[0059] In this case, the optical flow is determined based on the correspondence between the point cloud data set obtained by the vehicle-mounted radar and the image captured by the imaging device.

[0060] However, the point cloud data set is usually sparse, and accordingly, the optical flow determined based on the correspondence between the relatively sparse point cloud data set and the image is also sparse, that is, only sparse optical flow can be obtained by the prior art, and the sparse optical flow is difficult to meet the needs of application scenarios such as assisted driving or automatic driving.

[0061] Exemplary system

[0062] The embodiments of the present disclosure can be applied to application scenarios that need to determine optical flow, which can include application scenarios such as assisted driving or automatic driving.

[0063] The device for implementing the optical flow determination method of the embodiments of the present disclosure can be an electronic device such as a computer, an intelligent driving control device, or a server (such as a vehicle-mounted server), and in Figure 1 An example diagram of the device is disclosed in the

[0064] Referring to Figure 1 The device 100 for implementing the optical flow determination method of the embodiments of the present disclosure is applied to application scenarios of assisted driving or automatic driving, and the device 100 can be connected with the imaging device 200 and the sensor 300 respectively, and interact with information.

[0065] The connection between the device 100 and the imaging device 200 and the sensor 300 is not limited to physical connection. In one example, the connection between the device 100 and the imaging device 200 and the sensor 300 can be electrical connection. Alternatively, in another example, the connection between the device 100 and the imaging device 200 and the sensor 300 can be network connection, i.e., the device 100 can interact with the imaging device 200 and the sensor 300 respectively through the network.

[0066] The sensor 300 is a device capable of generating a point cloud dataset. For example, the sensor 300 can include a lidar, in which case the point cloud dataset can include a point cloud dataset generated by the lidar according to laser; or the sensor 300 can include an imaging processing device that generates a point cloud dataset according to an image, in which case the point cloud dataset can include a point cloud dataset generated by the imaging processing device according to the image.

[0067] In addition, after generating the point cloud dataset corresponding to the target road, the sensor 300 can transmit the point cloud dataset corresponding to the target road to the device 100. The point cloud dataset includes a plurality of points, each of which refers to a corresponding point cloud data. In one example, the sensor 300 is a lidar, and the laser emitted by the lidar is reflected after encountering a certain position of an object on the road, so each point (i.e., point cloud data) in the point cloud dataset generally includes the three-dimensional coordinates of the certain position of the object and the laser reflection intensity.

[0068] The imaging device 200 can capture an image corresponding to the target road and transmit the image to the device 100, so that the device 100 obtains the image corresponding to the target road.

[0069] The point cloud dataset corresponding to the target road refers to a point cloud dataset generated based on the objects existing on the target road. If the point cloud dataset is generated by a lidar, the point cloud dataset can be generated by the laser emitted by the lidar to the objects on the target road. The image corresponding to the target road refers to the image containing the target road obtained after the imaging device captures the target road.

[0070] After receiving the point cloud dataset and the image, the device 100 can perform the light flow determination method provided by the embodiments of the present disclosure to determine the corresponding light flow. Specifically, the device 100 determines the ground truth of the first light flow according to the received point cloud dataset and the image, where the first light flow can be a forward light flow or a reverse light flow. Then, the device 100 determines the predicted value of the first light flow based on the ground truth of the first light flow, where the predicted value of the first light flow is denser than the ground truth of the first light flow. The device 100 determines the predicted value of the second light flow based on the predicted value of the first light flow and the correspondence between the first light flow and the second light flow. Since the predicted value of the first light flow is denser, the predicted value of the second light flow determined by the device 100 is also denser. That is, by performing the light flow determination method provided by the embodiments of the present disclosure, the device 100 can determine the predicted value of the first light flow and the predicted value of the second light flow, which are denser.

[0071] If the embodiments of the present disclosure are applied to the application scenarios of assisted driving or automatic driving, the imaging device 200 can include a vehicle-mounted camera, the sensor 300 can include a vehicle-mounted sensor, and the device 100 can be a server (for example, a vehicle-mounted intelligent terminal) in a vehicle. In this case, the server in the vehicle can determine the dense light flow based on the scheme provided by the embodiments of the present disclosure to meet the needs of the application scenarios of assisted driving or automatic driving.

[0072] Example method 1

[0073] Figure 2 FIG. 1 is a flowchart of a light flow determination method provided by an example embodiment of the present disclosure. The embodiment can be applied to an electronic device, such as a server in a vehicle. Figure 2 As shown in FIG. 1, the method includes the following steps:

[0074] In step S201, the ground truth of the first light flow is determined based on the point cloud dataset and the image corresponding to the target road.

[0075] The first light flow can be a forward light flow or a reverse light flow. The forward light flow is the light flow from the perspective of the imaging device that captures the image to the virtual perspective. The reverse light flow is the light flow from the virtual perspective to the perspective of the imaging device.

[0076] The virtual perspective can include different types of perspectives. For example, in the application scenarios of assisted driving or automatic driving, the virtual perspective is usually a bird-eye-view (bev) perspective. In other application scenarios, the virtual perspective can be the perspective of the left view of the vehicle or other perspectives.

[0077] In the embodiments of the present disclosure, the ground truth of the first optical flow can be determined based on the coordinates (i.e., the first coordinates) of each point in the point cloud dataset projected on the image and the coordinates (i.e., the second coordinates) of the point projected on the virtual perspective plane. If the first optical flow is a forward optical flow, the ground truth of the first optical flow can be the difference between the second coordinates and the first coordinates of the same point in the point cloud dataset. If the first optical flow is a reverse optical flow, the ground truth of the first optical flow can be the difference between the first coordinates and the second coordinates of the same point in the point cloud dataset.

[0078] The first coordinates of each point in the point cloud dataset can be determined based on the first extrinsic parameter of the sensor relative to the imaging device. The sensor is used to obtain the point cloud dataset corresponding to the target road and transmit the point cloud dataset to the electronic device for executing the optical flow determination method provided by the present disclosure. The imaging device is used to obtain the image corresponding to the target road and transmit the image to the electronic device for executing the optical flow determination method provided by the present disclosure. The first extrinsic parameter of the sensor relative to the imaging device is a parameter representing the coordinate system conversion relationship between the coordinates of the sensor and the imaging device, such as a rotation matrix and / or a translation matrix representing the coordinate system conversion relationship between the coordinates of the sensor and the imaging device.

[0079] Correspondingly, the second coordinates of each point in the point cloud dataset projected on the virtual perspective plane can be determined based on the second extrinsic parameter of the sensor relative to the imaging device of the virtual perspective. The second extrinsic parameter is a parameter representing the coordinate system conversion relationship between the coordinates of the sensor and the imaging device of the virtual perspective, such as a rotation matrix and / or a translation matrix representing the coordinate system conversion relationship between the coordinates of the sensor and the imaging device of the virtual perspective.

[0080] In step S202, the predicted value of the first optical flow is determined based on the ground truth of the first optical flow, and the predicted value of the first optical flow is denser than the ground truth of the first optical flow.

[0081] In the embodiments of the present disclosure, the predicted value of the first optical flow can be determined based on a neural network model. The neural network model has self-learning ability, and in this case, the neural network model can learn the predicted value of the first optical flow based on the ground truth of the first optical flow.

[0082] In step S203, the predicted value of the second optical flow is determined based on the predicted value of the first optical flow and the correspondence between the first optical flow and the second optical flow.

[0083] The second optical flow is different from the first optical flow. If the first optical flow is a forward optical flow, the second optical flow is a reverse optical flow; if the first optical flow is a reverse optical flow, the second optical flow is a forward optical flow. Moreover, the prediction value of the first optical flow is dense, and the prediction value of the second optical flow is determined based on the prediction value of the first optical flow and the correspondence between the first optical flow and the second optical flow, so the prediction value of the second optical flow is also dense.

[0084] When the optical flow is determined by the scheme provided in the embodiments of the present disclosure, after the true value of the first optical flow is determined, the prediction value of the first optical flow is determined based on the true value of the first optical flow, and the prediction value of the second optical flow is determined based on the prediction value of the first optical flow and the correspondence between the first optical flow and the second optical flow. The prediction value of the first optical flow is denser than the true value of the first optical flow, so the scheme provided in the embodiments of the present disclosure can determine the prediction value of the first optical flow which is dense. Further, since the prediction value of the first optical flow is dense, the prediction value of the second optical flow determined is also dense. That is, the scheme provided in the embodiments of the present disclosure can obtain dense first optical flow and second optical flow, and solve the problem that the prior art can only obtain sparse optical flow.

[0085] Referring to Figure 3 In another exemplary embodiment of the present disclosure, based on the above Figure 2 The method further includes the following steps:

[0086] In step S204, the true value of the second optical flow is determined based on the point cloud data set and the image.

[0087] The point cloud data set and the image can be the point cloud data set and the image applied in step S201, and the point cloud data set and the image can be acquired in the same way as in step S201, which will not be described herein.

[0088] In addition, in the Figure 3 In actual application scenarios, the execution time of step S204 is not strictly limited. For example, step S204 can be executed simultaneously with step S201, or step S202, or step S203, or step S204 can be executed between step S201 and step S202, or between step S202 and step S203.

[0089] In step S205, the evaluation result of the prediction value of the second optical flow is determined based on the true value of the second optical flow and the prediction value of the second optical flow.

[0090] Since the prediction value of the second optical flow is based on the prediction value of the first optical flow, and the correspondence between the first optical flow and the second optical flow is determined, the prediction value of the second optical flow may be incorrect. In this case, the evaluation result of the prediction value of the second optical flow can be determined based on step S205, so that whether the prediction value of the second optical flow is accurate can be determined according to the evaluation result.

[0091] In a feasible example, the operation of determining the evaluation result of the prediction value of the second optical flow disclosed in step S205 can be implemented by the following steps:

[0092] First, based on the true value of the second optical flow corresponding to the same coordinate and the prediction value of the second optical flow, an accuracy parameter is calculated.

[0093] The accuracy parameter is the ratio of the first quantity to the second quantity, the first quantity is the number of times that the true value of the second optical flow is the same as the prediction value of the second optical flow, and the second quantity is the reference quantity of the true value of the second optical flow used for comparison.

[0094] In the embodiments of the present disclosure, the accuracy parameter is calculated based on the true value of the second optical flow corresponding to the same coordinate and the prediction value of the second optical flow. The coordinate can include a first coordinate of a point in the point cloud data set on the image projection, or can include a second coordinate of the point in the point cloud data set projected on the virtual perspective plane.

[0095] In an example, the coordinate is the second coordinate, the virtual perspective is the bev perspective, the second coordinate of a certain point in the point cloud data set projected on the bev perspective plane is (3, 1), the true value of the second optical flow corresponding to the coordinate is (-1, 1), and the prediction value of the second optical flow corresponding to the coordinate is also (-1, 1). In this case, it can be considered that the true value of the second optical flow corresponding to the coordinate is the same as the prediction value of the second optical flow.

[0096] In addition, the accuracy parameter is the ratio of the first quantity to the second quantity. For example, if the first quantity is 8 and the second quantity is 10, that is, the reference quantity of the true value of the second optical flow used for comparison is 10, and in 10 comparisons, the number of times that the true value of the second optical flow is the same as the prediction value of the second optical flow is 8. In this case, it can be determined that the accuracy parameter is 80%.

[0097] Then, based on the accuracy parameter and a preset first threshold, the evaluation result of the prediction value of the second optical flow is determined.

[0098] In the scheme provided in the embodiments of the present disclosure, the evaluation result of the prediction value of the second optical flow is determined by comparing the accuracy parameter and the first threshold. If the accuracy parameter is greater than the first threshold, it can be determined that the evaluation result of the prediction value of the second optical flow is accurate, and if the accuracy parameter is less than or equal to the first threshold, it can be determined that the evaluation result of the prediction value of the second optical flow is inaccurate.

[0099] In addition, the specific value of the first threshold can be set according to actual needs. In general, the higher the accuracy requirement for the predicted value of the second optical flow is, the higher the specific value of the first threshold tends to be.

[0100] Referring to Figure 4 In another example embodiment of the present disclosure, the above-mentioned Figure 2 On the basis of the embodiment shown, the following steps can also be included:

[0101] In step S206, when it is determined that the evaluation result of the predicted value of the second optical flow meets the preset condition, the image under the virtual perspective is determined based on the predicted value of the second optical flow and the image.

[0102] In the scheme provided in the embodiments of the present disclosure, the evaluation result of the predicted value of the second optical flow meeting the preset condition generally means that the predicted value of the second optical flow is greater than the preset first threshold, in which case the predicted value of the second optical flow can be considered to be accurate.

[0103] In addition, the optical flow can reflect the displacement change amount of the pixel coordinates of the image captured by the imaging device and the image under the virtual perspective. Therefore, after it is determined that the predicted value of the second optical flow is correct, the image under the virtual perspective can be determined based on the predicted value of the second optical flow.

[0104] In a feasible design, the operation can be implemented by the following steps:

[0105] In the first step, the correspondence between the first pixel and the second pixel is determined based on the predicted value of the second optical flow, wherein the first pixel is a pixel in the image under the imaging device perspective, and the second pixel is a pixel in the image under the virtual perspective.

[0106] The image under the imaging device perspective can include the image corresponding to the target road in step S201.

[0107] Since the predicted value of the second optical flow reflects the displacement change amount of the pixel coordinates of the image under the imaging device perspective and the image under the virtual perspective, the coordinates of the corresponding second pixel can be determined based on the predicted value of the second optical flow and the coordinates of the first pixel.

[0108] For example, the second optical flow is the reverse optical flow flow(u, v), the coordinates of a certain first pixel are (u1, v1), and the coordinates of the second pixel corresponding to the first pixel are (u2, v2). If the value of each dimension in flow(u, v) is an integer (which can include positive and / or negative numbers), the correspondence between the first pixel and the second pixel can be represented by the following formula:

[0109] (u1, v1) = flow(u, v) + (u2, v2).

[0110] Further, sometimes the prediction value of the second optical flow includes a floating point number, i.e. the value of at least one dimension in flow(u, v) includes a floating point number, in which case, interpolation can be performed on flow(u, v), and the optical flow after interpolation is flow(u0, v0), and u0 and v0 are both integers. In this case, the correspondence between the first pixel and the second pixel can be represented by the following formula:

[0111] (u1, v1) = flow(u0, v0) + (u2, v2).

[0112] Secondly, based on the correspondence between the first pixel and the second pixel, and the image of the imaging device perspective, the image under the virtual perspective is determined.

[0113] In this step, based on the correspondence between the first pixel and the second pixel, and the features contained in the image of the imaging device perspective, the features of the second pixel corresponding to the first pixel can be determined, and then the image under the virtual perspective can be determined.

[0114] In step S203, the operation of determining the prediction value of the second optical flow based on the prediction value of the first optical flow, and the correspondence between the first optical flow and the second optical flow is disclosed. Referring to Figure 5 In another exemplary embodiment of the present disclosure, based on the above Figure 2 The operation can include the following steps based on the embodiment shown in the figure:

[0115] In step S2031, the coordinates of the prediction value of the second optical flow are determined based on the prediction value of the first optical flow corresponding to each coordinate.

[0116] Wherein, since the first optical flow is a forward optical flow or a reverse optical flow, the second optical flow is different from the first optical flow, and the coordinates of the first optical flow and the second optical flow corresponding to the same point in the point cloud data set in the same dimension are opposite numbers. For example, if the first optical flow corresponding to a certain point in the point cloud data set is (a, b), the second optical flow corresponding to the point is (-a, -b).

[0117] In addition, the coordinates of a certain pixel in the image of the imaging device perspective are set as the third coordinates (u3, v3), the coordinates of the pixel converted to the image under the virtual perspective are the fourth coordinates (u4, v4), and the reverse optical flow corresponding to the pixel is (-a1, -b1), and the forward optical flow corresponding to the pixel is (a1, b1), then according to the meaning of the forward optical flow and the reverse optical flow, the following formula can be obtained:

[0118] (u4, v4) + (-a1, -b1) = (u3, v3) Formula (1);

[0119] (u3, v3) + (a1, b1) = (u4, v4) Equation (2).

[0120] For example, if the coordinates of a pixel in the image of the imaging device perspective is (2, 2), the coordinates of the pixel in the image of the virtual perspective is (3, 1), the forward optical flow corresponding to the pixel is (1, -1), and the backward optical flow corresponding to the pixel is (-1, 1), that is, the following equation is established:

[0121] (3, 1) + (-1, 1) = (2, 2);

[0122] (2, 2) + (1, -1) = (3, 1).

[0123] Wherein, if the first optical flow is the backward optical flow, based on Equation (1), the coordinates (u3, v3) of the predicted value of the forward optical flow corresponding to a pixel in the image of the virtual perspective can be determined according to the coordinates (u4, v4) of the pixel and the backward optical flow (-a1, -b1) corresponding to the pixel. If the first optical flow is the forward optical flow, based on Equation (2), the coordinates (u4, v4) of the predicted value of the backward optical flow corresponding to a pixel in the image of the imaging device perspective can be determined according to the coordinates (u3, v3) of the pixel and the forward optical flow (a1, b1) corresponding to the pixel.

[0124] In step S2032, the predicted value of the second optical flow is determined based on the coordinates of the predicted value of the second optical flow and the correspondence relationship between the first optical flow and the second optical flow.

[0125] Wherein, the correspondence relationship between the first optical flow and the second optical flow generally includes that the coordinates of the first optical flow and the second optical flow corresponding to the same pixel in the same dimension are opposite numbers. In this case, based on the predicted value of the first optical flow and the coordinates of the predicted value of the second optical flow determined through step S2031, the predicted value of the second optical flow corresponding to each coordinate can be determined.

[0126] Through the operations of step S2031 and step S2032, the predicted value of the second optical flow can be determined based on the predicted value of the first optical flow and the correspondence relationship between the first optical flow and the second optical flow. In this scheme, the predicted value of the first optical flow is relatively dense, and correspondingly, the predicted value of the second optical flow determined is also relatively dense, thereby meeting the demand of obtaining dense optical flow.

[0127] Example method 2

[0128] Figure 6 is a flowchart of a model training method for determining optical flow provided by an example embodiment of the present disclosure, as shown in Figure 6 The method includes the following steps:

[0129] Step S211, determine the ground truth of the first optical flow based on the optical flow determination method provided in any one of the embodiments of the example method 1 of the present disclosure.

[0130] The ground truth of the first optical flow is determined based on the point cloud data set and the image corresponding to the target road, and the first optical flow is a forward optical flow or a reverse optical flow. The forward optical flow is an optical flow from the perspective of the imaging device that captures the image to the virtual perspective, and the reverse optical flow is an optical flow from the virtual perspective to the perspective of the imaging device. The optical flow determination method provided in any one of the embodiments of the example method 1 of the present disclosure is a method for determining the optical flow corresponding to any one of the schematic diagrams. Figures 2 to 5

[0131] Step S212, process the point cloud data set and the image using the neural network model to be trained to obtain the predicted value of the first optical flow.

[0132] The neural network model has a deep learning function and can realize optical flow prediction. Based on the neural network model, the prediction of the first optical flow can be realized, and thus the predicted value of the first optical flow is obtained.

[0133] Step S213, determine the network loss based on the ground truth of the first optical flow and the predicted value of the first optical flow.

[0134] In this step, the network loss can be determined by existing deep learning technology. For example, the ground truth of the first optical flow and the predicted value of the first optical flow are both two-dimensional matrices. The network loss can be determined according to the difference between the two two-dimensional matrices.

[0135] Step S214, train the neural network model based on the network loss.

[0136] In the process of training the neural network model based on the network loss, the parameters of the neural network model can be adjusted based on the network loss, and after adjusting the parameters each time, the network loss is determined again through the neural network model after adjusting the parameters, until the network loss is less than a preset threshold, or the number of times of adjusting the parameters of the neural network model reaches a preset number.

[0137] If the network loss is less than the preset threshold, it usually indicates that the accuracy of the predicted value of the first optical flow determined by the neural network model is high, and in this case, the training of the neural network model can be ended.

[0138] In addition, if the number of times of adjusting the parameters of the neural network model reaches the preset number, it usually indicates that the parameters of the neural network model have been adjusted many times in the process of training the neural network model. In order to save computing power, the training of the neural network model can also be ended.

[0139] ​The model training method for determining an optical flow provided by the embodiments of the present disclosure can be used to train a neural network model, so that the trained neural network model can be used to determine a predicted value of the first optical flow with high accuracy.

[0140] The method provided by the embodiments of the present disclosure trains the neural network model based on the true value of the first optical flow and the predicted value of the first optical flow, and the training process is simple and efficient, which facilitates efficient determination of the predicted value of the optical flow.

[0141] An exemplary device 1

[0142] Figure 7 is a structural diagram of an optical flow determination device provided by an exemplary embodiment of the present disclosure. The optical flow determination device can be arranged in a terminal device, a server, or other electronic device, or on an object such as a vehicle, to execute the optical flow determination method of any of the embodiments of the present disclosure. As shown in Figure 7 The optical flow determination device of this embodiment includes a first true value determination module 201, a first predicted value determination module 202, and a second predicted value determination module 203.

[0143] The first true value determination module 201 is configured to determine a true value of a first optical flow based on a point cloud data set and an image corresponding to a target road.

[0144] The first predicted value determination module 202 is configured to determine a predicted value of the first optical flow based on the true value of the first optical flow determined by the first true value determination module 201, and the predicted value of the first optical flow is denser than the true value of the first optical flow.

[0145] The second predicted value determination module 203 is configured to determine a predicted value of a second optical flow based on the predicted value of the first optical flow determined by the first predicted value determination module 202 and a corresponding relationship between the first optical flow and the second optical flow.

[0146] The first optical flow is a forward optical flow or a reverse optical flow. The forward optical flow is an optical flow from a perspective of an imaging device that captures the image to a virtual perspective, and the reverse optical flow is an optical flow from the virtual perspective to the perspective of the imaging device.

[0147] In some embodiments, referring to Figure 8 As shown in an example diagram of a structure of an optical flow determination device, the optical flow determination device can further include:

[0148] The second true value determination module 204 is configured to determine a true value of the second optical flow based on the point cloud data set and the image.

[0149] The evaluation result determination module 205 is configured to determine an evaluation result of the second optical flow based on the ground truth of the second optical flow determined by the second ground truth determination module 204 and the predicted value of the second optical flow determined by the second predicted value determination module 203.

[0150] In one possible implementation, the evaluation result determination module 205 includes:

[0151] The parameter determination unit 2051 is configured to calculate an accuracy parameter based on the ground truth of the second optical flow and the predicted value of the second optical flow corresponding to the same coordinate, where the accuracy parameter is a ratio of a first quantity and a second quantity, the first quantity is a number of times that the ground truth of the second optical flow is the same as the predicted value of the second optical flow, and the second quantity is a reference quantity of the ground truth of the second optical flow used for comparison.

[0152] The evaluation result determination unit 2052 is configured to determine the evaluation result of the predicted value of the second optical flow based on the accuracy parameter determined by the parameter determination unit 2051 and a preset first threshold.

[0153] In some embodiments, referring to the example diagram of the structure of the optical flow determination apparatus shown in Figure 8 The optical flow determination apparatus can further include:

[0154] The image determination module 206 is configured to determine an image under the virtual perspective based on the predicted value of the second optical flow and the image when it is determined that the evaluation result of the predicted value of the second optical flow determined by the evaluation result determination module 205 meets a preset condition.

[0155] In one possible implementation, referring to the example diagram of the structure of the optical flow determination apparatus shown in Figure 8 The image determination module 206 includes:

[0156] The correspondence determination unit 2061 is configured to determine a correspondence between a first pixel and a second pixel based on the predicted value of the second optical flow, where the first pixel is a pixel in the image under the perspective of the imaging device, and the second pixel is a pixel in the image under the virtual perspective.

[0157] The image determination unit 2062 is configured to determine the image under the virtual perspective based on the correspondence determined by the correspondence determination unit 2061 and the image under the perspective of the imaging device.

[0158] In some embodiments, referring to the example diagram of the structure of the optical flow determination apparatus shown in Figure 8 The second predicted value determination module 203 can include:

[0159] The coordinate determination unit 2031 is configured to determine the coordinates of the predicted value of the second optical flow based on the predicted value of the first optical flow corresponding to each coordinate.

[0160] The predicted value determination unit 2032 is configured to determine the predicted value of the second optical flow based on the coordinates of the predicted value of the second optical flow determined by the coordinate determination unit 2031 and the correspondence between the first optical flow and the second optical flow.

[0161] The optical flow determination apparatus provided by the embodiment of the present disclosure can determine the predicted value of the first optical flow based on the true value of the first optical flow, and determine the predicted value of the second optical flow based on the predicted value of the first optical flow and the correspondence between the first optical flow and the second optical flow. The predicted value of the first optical flow is denser than the true value of the first optical flow, and therefore, the apparatus provided by the embodiment of the present disclosure can determine the predicted value of the first optical flow which is denser. Further, since the predicted value of the first optical flow is denser, the predicted value of the second optical flow determined is also denser. That is, the apparatus provided by the embodiment of the present disclosure can obtain the first optical flow and the second optical flow which are denser, and solve the problem that the prior art can only obtain sparse optical flow.

[0162] An exemplary apparatus 2

[0163] Figure 9 is a structural diagram of a model training apparatus for determining optical flow provided by an exemplary embodiment of the present disclosure. The model training apparatus for determining optical flow can be arranged in a terminal device, a server, or the like electronic device, or an object such as a vehicle, and perform the model training method for determining optical flow of any one of the embodiments of the present disclosure. As shown in the figure, the model training apparatus for determining optical flow of the embodiment includes an optical flow true value determination module 301, an optical flow predicted value determination module 302, a network loss determination module 303, and a network model training module 304. Figure 9

[0164] The optical flow true value determination module 301 is configured to determine the true value of the first optical flow based on the optical flow determination method provided by any one of the embodiments, and the true value of the first optical flow is determined based on the point cloud data set and the image corresponding to the target road.

[0165] The optical flow predicted value determination module 302 is configured to process the point cloud data set and the image by using the neural network model to be trained to obtain the predicted value of the first optical flow.

[0166] The network loss determination module 303 is configured to determine the network loss based on the true value of the first optical flow determined by the optical flow true value determination module 301 and the predicted value of the first optical flow determined by the optical flow predicted value determination module 302.

[0167] ​The network model training module 304 is configured to train the neural network model based on the network loss determined by the network loss determination module 303.

[0168] The first optical flow is a forward optical flow or a reverse optical flow. The forward optical flow is an optical flow from a perspective of an imaging device that captures the image to the virtual perspective. The reverse optical flow is an optical flow from the virtual perspective to the perspective of the imaging device.

[0169] The model training apparatus for determining an optical flow provided by the embodiments of the present disclosure can train a neural network model, so that a prediction value of the first optical flow with high accuracy can be determined by the trained neural network model subsequently. Moreover, the apparatus has a simple process for training the neural network model, and has high training efficiency, which facilitates efficient determination of the prediction value of the optical flow.

[0170] An example electronic device

[0171] Hereinafter, an electronic device according to an embodiment of the present disclosure will be described with reference to Figure 10 FIG. 1. Figure 1 In an example embodiment of the present disclosure, the electronic device can be the device 100 shown in Figure 1 FIG. 1, or in another example embodiment of the present disclosure, the electronic device can be an electronic device including the device 100, the imaging device 200, and the sensor 300 shown in FIG. 1. Of course, the electronic device can also be in other forms, which are not limited by the present disclosure.

[0172] Figure 10 FIG. 1 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure.

[0173] As shown in Figure 10 FIG. 1, the electronic device 11 includes one or more processors 111 and a memory 112.

[0174] The processor 111 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 11 to perform desired functions.

[0175] The memory 112 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk drives, solid-state drives, and / or the like. The computer-readable storage media can store one or more computer program instructions executable by the processor 111 to implement the optical flow determination method or the model training method for determining an optical flow of various embodiments of the present application described above, and / or other desired functions. Various contents such as point cloud data sets and images can also be stored in the computer-readable storage media.

[0176] In one example, the electronic device 11 can further include an input device 113 and an output device 114, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0177] For example, the input device 113 can be a microphone or a microphone array, or the input device 113 can be a communication network connector, and the like.

[0178] In addition, the input device 113 can further include, for example, a keyboard, a mouse, and the like.

[0179] The output device 114 can output various information to the outside, including the predicted values of the first optical flow and the second optical flow, and the like. The output device 114 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0180] Of course, in order to simplify, Figure 10 Only some of the components in the electronic device 11 related to the present application are shown in FIG. 1, and components such as buses, input / output interfaces, and the like are omitted. In addition, the electronic device 11 can include any other appropriate components according to specific application cases.

[0181] Exemplary computer program product and computer-readable storage medium

[0182] In addition to the above-described methods and devices, embodiments of the present application can also be computer program products including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the optical flow determination method or the model training method for determining an optical flow according to various embodiments of the present application described in the above “Exemplary Methods” section of the specification.

[0183] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. The embodiments of the present application are not limited by the programming languages used to implement the methods described herein.

[0184] In addition, the embodiments of the present application can also be a computer readable storage medium, which stores computer program instructions, and the computer program instructions, when executed by a processor, cause the processor to perform the steps of the optical flow determination method or the model training method for determining the optical flow described in the "Exemplary Method" section of the present specification according to various embodiments of the present application.

[0185] The computer readable storage medium can take the form of one or more combinations of any type of readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0186] The above describes the basic principles of the present application in combination with specific embodiments, but it should be noted that the advantages, advantages, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, advantages, effects and the like cannot be considered as the must-have of each embodiment of the present application. In addition, the above specific details are only for the purpose of example and understanding, and the above details do not limit the present application to the must-use specific details.

[0187] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0188] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0189] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0190] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for determining optical flow, comprising: Based on the point cloud datasets and images corresponding to the target roads, the ground truth value of the first optical flow is determined; Based on the true value of the first optical flow, a predicted value of the first optical flow is determined, wherein the predicted value of the first optical flow is denser than the true value of the first optical flow. Based on the predicted value of the first optical flow and the correspondence between the first optical flow and the second optical flow, the predicted value of the second optical flow is determined. If the first optical flow is a forward optical flow, then the second optical flow is a reverse optical flow; if the first optical flow is a reverse optical flow, then the second optical flow is a forward optical flow. Wherein, the forward optical flow is the optical flow from the viewpoint of the imaging device that captures the image to the virtual viewpoint, and the reverse optical flow is the optical flow from the virtual viewpoint to the viewpoint of the imaging device; If the first optical flow is a reverse optical flow, the coordinates of the pixel in the image of the virtual viewpoint are determined according to the coordinates of the pixel in the image of the virtual viewpoint and the predicted value of the reverse optical flow corresponding to the pixel; if the first optical flow is a forward optical flow, the coordinates of the pixel in the image of the virtual viewpoint are determined according to the coordinates of the pixel in the image of the image of the imaging device and the predicted value of the forward optical flow corresponding to the pixel.

2. The method according to claim 1, wherein, Also includes: Based on the point cloud dataset and the image, the ground truth value of the second optical flow is determined; Based on the true value of the second optical flow and the predicted value of the second optical flow, the evaluation result of the predicted value of the second optical flow is determined.

3. The method according to claim 2, wherein, The evaluation result for determining the predicted value of the second optical flow includes: Based on the true value and predicted value of the second optical flow corresponding to the same coordinate, an accuracy parameter is calculated. The accuracy parameter is the ratio of a first quantity to a second quantity. The first quantity is the number of times the true value and predicted value of the second optical flow are the same, and the second quantity is the reference quantity of the true value of the second optical flow used for comparison. The evaluation result of the predicted value of the second optical flow is determined based on the accuracy parameter and the preset first threshold.

4. The method according to claim 2, wherein, Also includes: When the evaluation result of the predicted value of the second optical flow meets the preset conditions, the image under the virtual viewpoint is determined based on the predicted value of the second optical flow and the image.

5. The method according to claim 4, wherein, Determining the image under the virtual viewpoint based on the predicted value of the second optical flow and the image includes: Based on the predicted value of the second optical flow, the correspondence between the first pixel and the second pixel is determined, wherein the first pixel is a pixel in the image from the perspective of the imaging device, and the second pixel is a pixel in the image from the virtual perspective. Based on the correspondence and the image from the perspective of the imaging device, the image under the virtual perspective is determined.

6. A model training method for determining optical flow, wherein, include: Based on the optical flow determination method according to any one of claims 1-5, the ground truth value of the first optical flow is determined, wherein the ground truth value of the first optical flow is determined based on the point cloud dataset and the image corresponding to the target road, respectively. The point cloud dataset and the image are processed using a neural network model to be trained to obtain the predicted value of the first optical flow. The network loss is determined based on the true value and the predicted value of the first optical flow. The neural network model is trained based on the network loss. Wherein, the first optical flow is a forward optical flow or a reverse optical flow, the forward optical flow is the optical flow from the viewpoint of the imaging device that captures the image to the virtual viewpoint, and the reverse optical flow is the optical flow from the virtual viewpoint to the viewpoint of the imaging device.

7. An optical flow determining device, comprising: The first truth value determination module is used to determine the truth value of the first optical flow based on the point cloud dataset and image corresponding to the target road respectively; The first prediction value determination module is used to determine the prediction value of the first optical flow based on the truth value of the first optical flow determined by the first truth value determination module, wherein the prediction value of the first optical flow is denser than the truth value of the first optical flow. The second prediction value determination module is used to determine the prediction value of the second optical flow based on the prediction value of the first optical flow determined by the first prediction value determination module and the correspondence between the first optical flow and the second optical flow. If the first optical flow is a forward optical flow, then the second optical flow is a reverse optical flow; if the first optical flow is a reverse optical flow, then the second optical flow is a forward optical flow. Wherein, the forward optical flow is the optical flow from the viewpoint of the imaging device that captures the image to the virtual viewpoint, and the reverse optical flow is the optical flow from the virtual viewpoint to the viewpoint of the imaging device; The device is further configured to: if the first optical flow is a reverse optical flow, determine the coordinates of the pixel in the image of the imaging device's viewpoint based on the coordinates of the pixel in the image of the virtual viewpoint and the predicted value of the reverse optical flow corresponding to the pixel; and if the first optical flow is a forward optical flow, determine the coordinates of the pixel in the image of the virtual viewpoint based on the coordinates of the pixel in the image of the imaging device's viewpoint and the predicted value of the forward optical flow corresponding to the pixel.

8. A model training apparatus for determining optical flow, wherein, include: The optical flow truth value determination module is used to determine the truth value of a first optical flow based on the optical flow determination method according to any one of claims 1-5, wherein the truth value of the first optical flow is determined based on the point cloud dataset and the image corresponding to the target road, respectively. The optical flow prediction value determination module is used to process the point cloud dataset and the image using the neural network model to be trained to obtain the predicted value of the first optical flow. A network loss determination module is used to determine network loss based on the truth value of the first optical flow determined by the optical flow truth value determination module and the predicted value of the first optical flow determined by the optical flow prediction value determination module. A network model training module is used to train the neural network model based on the network loss determined by the network loss determination module; Wherein, the first optical flow is a forward optical flow or a reverse optical flow, the forward optical flow is the optical flow from the viewpoint of the imaging device that captures the image to the virtual viewpoint, and the reverse optical flow is the optical flow from the virtual viewpoint to the viewpoint of the imaging device.

9. A computer-readable storage medium storing a computer program for executing the optical flow determination method according to any one of claims 1-5 or the model training method for determining optical flow according to claim 6.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the optical flow determination method according to any one of claims 1-5 or the model training method for determining optical flow according to claim 6.

Images