Training method of homography flow estimation model, homography flow estimation method and device

Abstract

This disclosure provides a training method and apparatus for a homography flow estimation model, a method and apparatus for homography flow estimation based on monocular images, a computer-readable storage medium, and an electronic device. The training method includes: extracting features from a first sample image and a second sample image in the same image sequence using a model to be trained, obtaining feature data; performing prediction using the model to be trained based on the feature data, obtaining predicted homography flow data; determining a homography flow loss value based on the predicted homography flow data and a preset homography flow loss function; and adjusting the parameters of the model to be trained based on the homography flow loss value to obtain a homography flow estimation model. This disclosure enables the training of a homography flow estimation model, allowing for the prediction of ground-based homography flow data from image sequences captured by a monocular camera, significantly improving the efficiency and accuracy of homography flow data prediction.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method for a homography flow estimation model, an apparatus for a homography flow estimation model, a homography flow estimation method, a homography flow estimation apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Homography estimation can be applied to image correction, image alignment, image stitching, augmented reality, and visual localization. Common homography estimation is primarily used to describe the inter-frame mapping relationships of common planes of objects in a scene. For example, in assisted driving and autonomous driving applications, the ground plane is a primary plane. Homography estimation can align the ground plane within different frames based on the ground equations corresponding to different frames and the inter-frame camera poses, thus providing ground-aligned inter-frame data for subsequent tasks and offering better data input for more accurate autonomous driving perception.

[0003] Currently, commonly used homography estimation methods include the following two approaches:

[0004] 1. Traditional homography estimation methods can detect feature points on the ground, perform feature matching based on the feature points, and use the RANSAC (Random Sample Consensus) algorithm to eliminate erroneous matching points, finally obtaining ground-based homography estimation data.

[0005] 2. A homography estimation scheme based on deep learning can calculate the homography estimation data of an image sequence by predicting the relative offsets of the four corners of each image in the sequence. Summary of the Invention

[0006] The embodiments of this disclosure provide a training method and apparatus for a homography flow estimation model, a homography flow estimation method and apparatus based on monocular images, a computer-readable storage medium, and an electronic device.

[0007] The embodiments of this disclosure provide a training method for a homography flow estimation model. The method includes: extracting features from a first sample image and a second sample image in the same image sequence using a model to be trained, wherein the second sample image is located after the first sample image; predicting based on the feature data using the model to be trained, thereby obtaining predicted homography flow data representing the relative positional changes of ground regions in the first and second sample images; determining a homography flow loss value based on the predicted homography flow data and a preset homography flow loss function; and adjusting the parameters of the model to be trained based on the homography flow loss value until a preset training termination condition is met, thereby obtaining a homography flow estimation model.

[0008] According to another aspect of the present disclosure, a homography flow estimation method based on monocular images is provided. The method includes: using a pre-trained homography flow estimation model to extract features from a first image and a second image in the same image sequence to obtain feature data, wherein the second image is located after the first image; and based on the feature data, performing homography flow prediction by the homography flow estimation model to generate homography flow data representing the relative positional changes of ground regions in the first image and the second image.

[0009] According to another aspect of the present disclosure, a training apparatus for a homography flow estimation model is provided. The apparatus includes: a first extraction module, configured to extract features from a first sample image and a second sample image in the same image sequence using a model to be trained, to obtain feature data, wherein the second sample image is located after the first sample image; a first prediction module, configured to predict homography flow data representing the relative positional change of ground regions in the first sample image and the second sample image based on the feature data and using the model to be trained; a first determination module, configured to determine a homography flow loss value based on the predicted homography flow data and a preset homography flow loss function; and a training module, configured to adjust the parameters of the model to be trained based on the homography flow loss value until a preset training termination condition is met, thereby obtaining a homography flow estimation model.

[0010] According to another aspect of the present disclosure, a homography flow estimation apparatus based on monocular images is provided. The apparatus includes: a second extraction module, configured to extract features from a first image and a second image in the same image sequence using a pre-trained homography flow estimation model to obtain feature data, wherein the second image is located after the first image; and a second prediction module, configured to perform homography flow prediction based on the feature data using the homography flow estimation model to generate homography flow data representing the relative positional changes of ground regions in the first image and the second image.

[0011] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores a computer program for performing the training method of the homography flow estimation model or the homography flow estimation method based on monocular images described above.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the training method for the homography flow estimation model or the homography flow estimation method based on monocular images described above.

[0013] The training method for the homography flow estimation model provided in the above embodiments of this disclosure involves extracting features from the first and second sample images using the model to be trained. Based on the feature data, predicted homography flow data of the relative positional changes of the ground region is obtained. Then, based on the predicted homography flow data and the homography flow loss function, the homography flow loss value is determined. Finally, based on the homography flow loss value, the parameters of the model to be trained are adjusted to obtain the homography flow estimation model. This method achieves the training of the homography flow estimation model and the prediction of homography flow data. The trained homography flow estimation model can predict homography flow data based on the ground in image sequences captured by a monocular camera. Compared to homography estimation based on the entire image, the homography flow estimation model provided in this disclosure only predicts homography flow for the ground, making the prediction process more targeted and greatly improving the efficiency and accuracy of homography flow data prediction.

[0014] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 This is the system diagram to which this disclosure applies.

[0017] Figure 2 This is a flowchart illustrating a training method for a homography flow estimation model provided in an exemplary embodiment of this disclosure.

[0018] Figure 3 This is a flowchart illustrating a training method for a homography flow estimation model provided in another exemplary embodiment of this disclosure.

[0019] Figure 4 This is a flowchart illustrating a training method for a homography flow estimation model provided in another exemplary embodiment of this disclosure.

[0020] Figure 5 This is a flowchart illustrating a training method for a homography flow estimation model provided in another exemplary embodiment of this disclosure.

[0021] Figure 6 This is a flowchart illustrating a training method for a homography flow estimation model provided in another exemplary embodiment of this disclosure.

[0022] Figure 7This is a flowchart illustrating a training method for a homography flow estimation model provided in another exemplary embodiment of this disclosure.

[0023] Figure 8 This is a flowchart illustrating a homography flow estimation method based on monocular images provided in an exemplary embodiment of this disclosure.

[0024] Figure 9 This is a schematic diagram of the structure of a training apparatus for a homography flow estimation model provided in an exemplary embodiment of this disclosure.

[0025] Figure 10 This is a schematic diagram of the structure of a training apparatus for a homography flow estimation model provided in another exemplary embodiment of this disclosure.

[0026] Figure 11 This is a schematic diagram of the structure of a homography flow estimation device based on a monocular image provided in an exemplary embodiment of the present disclosure.

[0027] Figure 12 This is a schematic diagram of the structure of a homography flow estimation device based on a monocular image provided in another exemplary embodiment of this disclosure.

[0028] Figure 13 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0029] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0031] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0032] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0033] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0034] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0035] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0036] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0040] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0041] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0042] Application Overview

[0043] Homography estimation is primarily used to describe the inter-frame mapping relationships of common planes among objects in a scene. Current homography estimation schemes include traditional feature point matching-based schemes and deep learning-based schemes.

[0044] Feature-point matching-based schemes can obtain ground feature points and perform homography estimation and prediction through feature matching and filtering. However, the entire process is lengthy, computationally intensive, and inefficient, failing to meet the real-time operational requirements of business scenarios. Deep learning-based schemes generally can only predict homography data based on the entire map and cannot specify homography estimation data for the ground plane, resulting in lower accuracy in homography estimation for the ground.

[0045] Therefore, existing homography estimation schemes cannot meet the requirements of high efficiency, real-time performance, and high accuracy. This disclosure aims to propose a method for training a homography flow estimation model for the ground plane using image sequences captured by a monocular camera, thereby addressing the problems of existing schemes.

[0046] Exemplary System

[0047] Figure 1 An exemplary system architecture 100 is shown for a training method or apparatus for a homography flow estimation model to which embodiments of the present disclosure may be applied.

[0048] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0049] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as image processing applications, video processing applications, navigation applications, web browser applications, etc.

[0050] Terminal device 101 can be various electronic devices, including but not limited to mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers.

[0051] Server 103 can be a server that provides various services, such as a background image processing server that processes image sequences uploaded by terminal device 101. The background image processing server can use the received image sequences for model training, or use the trained model to perform homography flow estimation on the image sequences, etc.

[0052] It should be noted that the training method for the homography flow estimation model or the homography flow estimation method based on monocular images provided in the embodiments of this disclosure can be executed by the server 103 or by the terminal device 101. Accordingly, the training device for the homography flow estimation model or the homography flow estimation device based on monocular images can be set in the server 103 or in the terminal device 101.

[0053] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. If images do not need to be acquired remotely or image processing does not need to be performed remotely, the above system architecture may exclude the network and include only servers or terminal devices.

[0054] Exemplary methods

[0055] Figure 2 This is a schematic flowchart illustrating a training method for a homography flow estimation model provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices (such as...). Figure 1 On the terminal device 101 or server 103 shown, such as Figure 2 As shown, the method includes the following steps:

[0056] Step 201: Use the model to be trained to extract features from the first and second sample images in the same image sequence to obtain feature data.

[0057] The second sample image is located after the first sample image. For example, if the first sample image was taken at time t, then the second sample image following it could be taken after time t. It should be noted that the above image sequence can be the original image sequence captured by the camera, or it can be a sequence of multiple images extracted from the images included in the original image sequence.

[0058] Typically, the above image sequence is captured by a monocular camera, and the image sequence can be captured while the camera is in motion. For example, while a vehicle is in motion, a camera mounted on it captures images of the vehicle's surrounding environment, resulting in an image sequence.

[0059] The training model mentioned above can include deep neural networks of various structures. For example, the training model can include networks such as ResNet (Residual Network) and VarGNet (Variable Group Network). These networks can be used to extract features from the input image. During feature extraction, the first sample image and the second sample image can be merged on the RGB channels respectively, and then feature data can be extracted from the merged image.

[0060] When extracting feature data, network structures such as Unet can be used to extract feature data at different scales. For example, Unet downsamples 5 times, and the scale of the feature map is half of the original size each time. The resulting feature scales are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size, respectively.

[0061] Step 202: Based on the feature data, use the model to be trained to make predictions and obtain predicted homography stream data representing the relative positional changes of the ground regions in the first sample image and the second sample image.

[0062] As an example, the feature data downsampled five times by Unet is then upsampled several times by a factor of 2 before being fused with large-scale features, and so on, to obtain fused feature data at multiple scales. Typically, the fused feature data used for homography flow prediction can be selected from fused feature data at multiple scales. For example, fused feature data at 1 / 2 or 1 / 4 the size of the original image can be selected for homography flow prediction; using fused feature data at these scales results in good model prediction speed and performance.

[0063] Homography refers to the projection mapping of pixels from one plane to another. Homography stream data is a type of optical flow that represents the instantaneous velocity of the corresponding pixel of a spatially moving object on the imaging plane; that is, the relative displacement feature of the actual spatial point represented by pixel A in one image to pixel A' in another image. It should be noted that the moving object mentioned above is an object that moves relative to the imaging plane. For example, when the camera is moving relative to the ground, the ground in the image captured by the camera is a moving object relative to the imaging plane. In this embodiment, the predicted homography stream data represents the relative displacement feature of the ground region in the first and second sample images relative to the imaging planes of the first and second sample images. Through homography stream data, a mapping relationship between the ground in the first and second sample images can be established. This mapping relationship requires homography stream estimation, and the homography stream data is the relative positional relationship between pixels on the ground mapped to the same spatial point on the ground in the two images, obtained through homography stream estimation.

[0064] In this embodiment, the model to be trained may include a homography flow prediction network. The homography flow prediction network can determine the ground regions in the first sample image and the second sample image respectively based on the fused feature data, and then use existing homography estimation methods to determine the mapping relationship between the two ground regions, thereby obtaining homography flow prediction data.

[0065] Step 203: Determine the homography flow loss value based on the predicted homography flow data and the preset homography flow loss function.

[0066] The homography flow loss function is used to quantitatively calculate the error between the predicted homography flow data and the actual homography flow data. The homography flow loss value can represent the error between the predicted homography flow data and the actual homography flow data.

[0067] The homography loss function can include at least one of various forms of loss functions, such as supervised, unsupervised, and self-supervised. Specific forms of the homography loss function can be found in the following optional embodiments.

[0068] Step 204: Based on the homography flow loss value, adjust the parameters of the model to be trained until the preset training termination condition is met, and obtain the homography flow estimation model.

[0069] The model can be trained using machine learning methods. Specifically, during training, the electronic device can employ gradient descent and backpropagation to adjust the parameters of the model, gradually reducing the homography flow loss. The model after each parameter adjustment is used as the next training model. Training ends when a preset training termination condition is met, thus obtaining the homography flow estimation model. This preset training termination condition may include, but is not limited to, at least one of the following: training time exceeds a preset duration; the number of training iterations exceeds a preset number; or the homography flow loss value converges.

[0070] The method provided in the above embodiments of this disclosure extracts features from a first sample image and a second sample image using a model to be trained. Based on the feature data, it obtains predicted homography flow data of the relative positional changes of the ground region. Then, based on the predicted homography flow data and the homography flow loss function, it determines the homography flow loss value. Finally, based on the homography flow loss value, it adjusts the parameters of the model to be trained to obtain a homography flow estimation model. This achieves the training of the homography flow estimation model. The trained homography flow estimation model can predict homography flow data based on the ground in image sequences captured by a monocular camera. Compared to homography estimation based on the entire image, the homography flow estimation model provided in this disclosure only predicts homography flow for the ground, making the prediction process more targeted and greatly improving the efficiency and accuracy of homography flow data prediction.

[0071] In some alternative implementations, the homography flow loss function includes the homography flow unsupervised loss function, and the homography flow loss value includes the first homography flow loss value.

[0072] Based on this, such as Figure 3 As shown, step 203 may include the following sub-steps:

[0073] Step 2031: Determine the ground mask data corresponding to the second sample image.

[0074] The ground mask data is the ground representation data determined from the second sample image through semantic segmentation. Typically, in the second sample image, pixels belonging to the ground region are set to the number "1", and pixels not belonging to the ground region are set to the number "0", thus obtaining the ground mask data.

[0075] Step 2032: Based on the predicted homography stream data, the first sample image is mapped onto the image plane of the second sample image to obtain the mapped image.

[0076] Specifically, since the predicted homography stream data uses the ground as a reference to represent the mapping relationship between the ground in the first sample image and the second sample image, the pixels included in the first sample image can be mapped to the image plane of the second sample image based on this mapping relationship, thus obtaining the mapped image warp(i t-1 HF pre warp() represents the image mapping method, HF pre Indicates the prediction of homography stream data, i t-1 This represents the first sample image.

[0077] Step 2033: Based on the mapped image, the second sample image, and the ground mask data, determine the first homography flow loss value using the homography flow unsupervised loss function.

[0078] Specifically, the electronic device can determine the first homography flow loss value Lu according to the following formula (1). HF :

[0079]

[0080] Among them, mask g This represents ground mask data, α is the set weight, and SSIM() calculates the similarity between two images. t Represents the second sample image, i t -warp(i t-1 HF pre This indicates that the corresponding pixel values ​​of two images are subtracted (either RGB values ​​or grayscale values ​​can be subtracted).

[0081] This embodiment calculates the first homography flow loss value by setting an unsupervised loss function for homography flow, thereby realizing an unsupervised training method when training the homography flow estimation model. This eliminates the need for extensive pre-annotation of sample images, improves training efficiency, enriches training methods, and enhances the flexibility of training the homography flow estimation model. It also helps to combine with other loss functions to improve the prediction accuracy of the trained model.

[0082] In some alternative implementations, the homography flow loss function includes the homography flow supervision loss function, and the homography flow loss value includes the second homography flow loss value.

[0083] Based on this, step 203 may include:

[0084] Based on the actual homography flow data and predicted homography flow data corresponding to the second sample image, the second homography flow loss value is determined using the homography flow supervision loss function.

[0085] The aforementioned actual homography stream data can be data generated by pre-annotating the first and second sample images to represent the actual homography stream data. Typically, the ground regions included in the first and second sample images can be compared in advance, and the displacement of the projection points of corresponding points representing the same point on the actual ground in the two images can be measured to obtain the aforementioned actual homography stream data.

[0086] As an example, the second homography flow loss value Ls can be calculated according to the following formula (2). HF :

[0087] Ls HF =|HF gt -HF pre | (2)

[0088] Among them, HF gt Represents actual homography stream data, HF pre This indicates the prediction of homography stream data.

[0089] This embodiment, by setting a supervised homography loss function, can make the model's predicted data closer to the actual data during the training process based on actual homography stream data and predicted homography stream data, further enriching the training methods and helping to improve the prediction accuracy of the trained model when combined with other loss functions.

[0090] Further reference Figure 4 This illustrates a flowchart of yet another embodiment of the training method for a homography flow estimation model. Figure 4 As shown above, in the above Figure 2 Based on the illustrated embodiment, the following steps may be included before step 204:

[0091] Step 205: Based on the feature data, perform spatial detection using the model to be trained to generate spatial detection data.

[0092] Spatial detection data is used to characterize the three-dimensional spatial features of objects in the first and second sample images. Typically, spatial detection data can be generated by a spatial detection network included in the model to be trained. Feature data is a collection of feature maps at multiple scales; therefore, feature maps of a certain scale (e.g., the smallest scale) can be selected from these feature maps and input into the spatial detection network to obtain spatial detection data.

[0093] As an example, spatial detection data may include, but is not limited to, at least one of the following: inter-frame pose data of the first and second sample images, ground equation data, etc. Regarding the methods for generating the above two types of spatial detection data, please refer to the following optional embodiments.

[0094] Step 206: Determine the spatial detection loss value based on the spatial detection data and the preset spatial detection loss function.

[0095] In this embodiment, the spatial detection loss function is typically a supervised loss function. That is, during training, the electronic device determines the error between the pre-labeled actual spatial detection data and the calculated predicted spatial detection data based on the spatial detection loss function. This error can be represented by the spatial detection loss value calculated by the spatial detection loss function.

[0096] like Figure 4 As shown, step 204 may further include:

[0097] Step 2041: Adjust the parameters of the model to be trained based on the homography flow loss value and the spatial detection loss value.

[0098] Specifically, the homography flow loss function and the spatial detection loss function can be added together to obtain the sum of the homography flow loss value and the spatial detection loss value. During training, the electronic device can adjust the parameters of the model to be trained, so that the sum of the homography flow loss value and the spatial detection loss value gradually decreases. The model after each parameter adjustment is used as the model to be trained in the next training. When the preset training termination condition is met, the training ends, thereby training a homography flow estimation model.

[0099] The above Figure 4 The method provided in the corresponding embodiment performs spatial detection using the model to be trained, generating spatial detection data. Based on the spatial detection data and the spatial detection loss function, combined with the homography flow loss function, the model to be trained is trained. This allows for the introduction of more parameters during training, making the model's parameters more closely match the actual scene, which helps improve the prediction accuracy of the trained homography flow estimation model. Simultaneously, the trained homography flow estimation model can output spatial detection data, thus enriching the functionality of the homography flow estimation model and greatly expanding its applicable scenarios.

[0100] In some optional implementations, the spatial detection data includes first spatial detection data and second spatial detection data. The first spatial detection data may be inter-frame pose data representing the relative camera pose between a first sample image and a second sample image, and the second spatial detection data may be ground equation data representing the ground equation corresponding to the second sample image.

[0101] like Figure 5 As shown, step 205 above may include:

[0102] Step 2051: Based on the feature data, the model to be trained performs inter-frame pose prediction to obtain first spatial detection data representing the relative pose of the camera between the first sample image and the second sample image.

[0103] The inter-frame pose prediction data can be obtained by the inter-frame pose network included in the model to be trained, using feature data for computation. The inter-frame pose network can perform computation on the feature data based on its network parameters to obtain the first spatial detection data representing the inter-frame pose data. It should be noted that existing techniques can be used to calculate inter-frame pose using neural networks, and will not be elaborated upon here.

[0104] Step 2052: Based on the feature data, the ground equation is predicted by the model to be trained to obtain the second spatial detection data representing the ground equation corresponding to the second sample image.

[0105] The ground equation data can be obtained by the ground equation prediction network included in the model to be trained, using feature data. The ground equation data is used to characterize the plane equation within the space where the ground region in the second sample image is located. Typically, the ground equation data can include ground normal vectors. From the ground normal vectors, according to representations such as point normal form or normal form, the second spatial detection data representing the ground equation can be obtained.

[0106] The ground equation prediction network can determine the positions of the points in three-dimensional space corresponding to the pixels in the ground region of the second sample image based on feature data. Based on the positions of these three-dimensional points, ground equation data including ground normal vectors can be obtained according to existing methods for determining ground normal vectors.

[0107] This embodiment provides two methods for generating spatial detection data. These two types of spatial detection data can be combined with homography flow data to train the entire model. Because the training process incorporates more diverse data, the trained homography flow estimation model can more comprehensively analyze the spatial features of the input image during prediction, thus improving the prediction accuracy. Simultaneously, the homography flow estimation model can also perform inter-frame pose prediction and ground equation prediction on the input image during prediction using an inter-frame pose network and a ground equation prediction network, enriching the functionality of the homography flow estimation model and expanding its application scenarios.

[0108] In some alternative implementations, the spatial detection loss function includes a supervised inter-frame pose loss function and a supervised ground equation loss function, and the spatial detection loss value includes a first spatial detection loss value and a second spatial detection loss value.

[0109] like Figure 5 As shown, step 206 above includes:

[0110] Step 2061: Based on the first spatial detection data and the corresponding actual inter-frame pose data, determine the first spatial detection loss value using the supervised inter-frame pose loss function.

[0111] The aforementioned actual inter-frame pose data can be generated by pre-annotating the first and second sample images to represent the camera's actual inter-frame pose. Alternatively, the actual inter-frame pose data can be set by pre-measuring the actual scene captured by the first and second sample images. It can also be obtained using other inter-frame pose evaluation methods (such as LOAM (Lidar Odometry and Mapping in Real-time) mapping methods).

[0112] Specifically, the supervised inter-frame pose loss function can be expressed by the following equation (3):

[0113] Ls pose =|p gt -p pre | (3)

[0114] Among them, Ls pose p represents the first-space detection loss value. gt p represents the actual inter-frame pose data, i.e., the true inter-frame pose value. pre This represents the predicted inter-frame pose data, i.e., the first spatial detection data.

[0115] Step 2062: Based on the second space detection data and the corresponding actual ground equation data, determine the second space detection loss value using the supervised ground equation loss function.

[0116] The aforementioned actual ground equation data can be data generated by pre-annotating the second sample image to represent the actual ground equation. Typically, the ground can be segmented from the scene image manually or based on existing object detection methods, and then the plane equation is fitted in the camera coordinate system to obtain the aforementioned actual ground plane equation.

[0117] Specifically, the supervised ground equation loss function can be expressed by the following equation (4):

[0118] Ls ground =|g gt -g pre | (4)

[0119] Among them, Ls ground G represents the second-space detection loss value. gt This represents the actual ground equation data, i.e., the true value of the ground equation, g. pre This represents the predicted ground equation data, i.e., the second space detection data.

[0120] Typically, during training, the supervised inter-frame pose loss function and the supervised ground equation loss function can be added to the homography flow loss function mentioned above. By adjusting the parameters of the model to be trained, the sum of the loss values ​​of each loss function is gradually reduced until convergence, thus realizing joint training based on spatial detection and homography evaluation.

[0121] The supervised inter-frame pose loss function and supervised ground equation loss function provided in this embodiment can enable the trained homography flow estimation model to output first and second spatial detection data that are closer to the actual situation, thereby improving the accuracy of the homography flow estimation model in spatial detection of images.

[0122] In some alternative implementations, such as Figure 6 As shown, step 205 above includes:

[0123] Step 2053: Determine the feature map of the target scale from the feature data.

[0124] Feature data is typically a set of data composed of multiple feature maps of different sizes. The feature data included in the feature map is usually in the form of a matrix, and the scale of the feature map is the number of rows and columns of the feature map.

[0125] The target scale can be a pre-set fixed scale, or it can be a scale determined from the feature data according to set rules. For example, the target scale can be the smallest scale among the scales of various feature maps.

[0126] As an example, the feature fusion network included in the model to be trained downsampled 5 times. After each downsampling, the scale of the feature map obtained will be 1 / 2 of the original scale. The resulting feature sizes are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image scale, respectively. Therefore, the feature map with the smallest scale is 1 / 32 of the original image scale. So the target scale here is related to the input image size and the number of downsampling times of the feature fusion network.

[0127] Step 2054: Based on the feature map at the target scale, perform spatial detection using the model to be trained to generate spatial detection data.

[0128] Specifically, the feature map at the target scale can be input into the aforementioned spatial detection network, which will then generate spatial detection data.

[0129] This embodiment determines the feature map of the target scale from the feature data, which allows for flexible selection of the receptive field of the feature map used for spatial detection. For example, when the target scale is the smallest scale, the receptive field of the feature map of the smallest scale is the largest, and the output spatial detection data is globally correlated with the image, thereby improving the accuracy of spatial detection.

[0130] In some alternative implementations, the homography flow loss function includes the homography flow self-supervised loss function, and the homography flow loss value includes a third homography flow loss value.

[0131] like Figure 7 As shown, step 203 includes:

[0132] Step 2034: Based on the first spatial detection data, determine the rotation matrix and translation matrix of the second sample image relative to the first sample image.

[0133] The rotation matrix and translation matrix can be included in the first spatial detection data, that is, the relative pose of the camera between the first sample image and the second sample image can be represented by the rotation matrix and translation matrix.

[0134] Step 2035: Based on the second spatial detection data, determine the ground normal vector and the height of the camera used to capture the image sequence.

[0135] Since the second spatial detection data is used to represent the ground equation corresponding to the second sample image, and the ground equation can be represented in normal form, the ground normal vector can be directly determined from the second spatial detection data. Because the ground equation is established in the camera coordinate system, and the origin of the camera coordinate system is the camera's position, the camera's height relative to the ground can be determined based on the ground plane equation.

[0136] Step 2036: Determine the homography matrix based on the ground normal vector, preset camera intrinsic parameters, height, rotation matrix, and translation matrix.

[0137] Among them, the camera intrinsic parameters can be pre-calibrated and are used to characterize the correspondence between points in the camera coordinate system and points in the image coordinate system.

[0138] The homography matrix H can be determined using the following formula (5):

[0139]

[0140] Where K represents the camera intrinsic parameters, K -1 R represents the inverse matrix of the camera intrinsic parameters, R represents the rotation matrix, t represents the translation matrix, and N represents the ground normal vector. T d represents the transpose of the ground normal vector. c Indicates the camera height.

[0141] Step 2037: Using the homography matrix, determine the actual homography stream data of the second sample image relative to the first sample image.

[0142] The homography matrix described above represents the mapping relationship between the pixels in the ground region of the first and second sample images. Therefore, the coordinate vectors of the pixels in the ground region of the first sample image can be multiplied by the homography matrix to obtain the mapped coordinates in the second sample image. Subtracting the corresponding mapped coordinates from the coordinates of the pixels in the ground region of the second sample image yields the actual homography stream data.

[0143] Step 2038: Based on the predicted homography flow data and the actual homography flow data, determine the third homography flow loss value using the homography flow self-supervised loss function.

[0144] Specifically, the third homography flow loss value Ls can be determined according to the homography flow self-monitored loss function expressed by the following formula (6). GHF :

[0145] Ls GHF =|HF pre -HF cal | (6)

[0146] Among them, HF pre This indicates predicted homography stream data, HF cal This represents actual homography streaming data. During model training, as the parameters of the model being trained are continuously adjusted, Ls... GHF The gap between the predicted homography stream data and the actual homography stream data gradually decreases.

[0147] Typically, when training a model, all the loss functions described in the above embodiments can be summed. By adjusting the parameters of the model to be trained, the sum of the loss values ​​of each loss function can be gradually reduced until convergence, thereby obtaining the trained homography flow estimation model.

[0148] This embodiment, by setting a self-supervised loss function for homography flow, can automatically calculate the actual homography flow data when training the homography flow estimation model, without the need for extensive pre-annotation of sample images. This improves training efficiency, enriches training methods, and enhances the flexibility of training the homography flow estimation model. It also helps to combine with other loss functions to improve the prediction accuracy of the trained model.

[0149] Figure 8 This is a schematic flowchart illustrating a homography flow estimation method based on monocular images provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices (such as...). Figure 1On the terminal device 101 or server 103 shown, such as Figure 8 As shown, the method includes the following steps:

[0150] Step 801: Using a pre-trained homography flow estimation model, feature extraction is performed on the first and second images in the same image sequence to obtain feature data.

[0151] Among them, the homography flow estimation model is based on the above. Figure 2 The image is obtained by training according to the method described in the corresponding embodiment. The second image is located after the first image. For example, if the second sample image was taken at the current time (e.g., time t), then the first image could have been taken before the current time.

[0152] The homography flow estimation model described above can include any suitable deep neural network structure. For example, the model to be trained can include networks such as ResNet (Residual Network) and VarGNet (Variable Group Network), which can be used to extract features from the input image. During feature extraction, the first and second images can be merged on the RGB channels respectively, and then feature data can be extracted from the merged image.

[0153] Step 802: Based on the feature data, homography flow prediction is performed by the homography flow estimation model to generate homography flow data representing the relative positional changes of ground regions in the first and second images.

[0154] Typically, before performing homography flow prediction, feature data can be fused at multiple scales to obtain fused feature data. This fused feature data is then input into the homography flow prediction network. The homography flow prediction network can determine the ground regions in the first and second images based on the fused feature data. Then, using existing homography estimation methods, the mapping relationship between the two ground regions is determined, thus obtaining the homography flow prediction data.

[0155] As an example, feature data can be obtained by downsampling five times from a deep neural network with a Unet structure, then upsampling it several times by a factor of 2 before fusing it with larger-scale features, and so on, to obtain fused feature data at multiple scales. Typically, the fused feature data used in homography flow prediction can be selected from fused feature data at multiple scales. For example, fused feature data at 1 / 2 or 1 / 4 the size of the original image can be selected for homography flow prediction; using fused feature data at these scales results in good model prediction speed and performance.

[0156] It should be understood that the homography flow estimation model used in this embodiment is different from that described above. Figure 2The homography flow estimation model trained in the corresponding embodiments is the same. For a description of the structure and prediction process of the homography flow estimation model, please refer to the above. Figure 2 The content described in the corresponding embodiments will not be repeated here.

[0157] The method provided in the above embodiments of this disclosure extracts features from the first and second images in the same image sequence using a homography flow estimation model to obtain feature data. Then, based on the feature data, the homography flow estimation model performs homography flow prediction to generate homography flow data representing the relative positional changes of ground regions in the first and second images. Since the model training uses image sequences captured by a monocular camera to predict homography flow data based on the ground, the prediction process of the model is more targeted, thereby greatly improving the efficiency and accuracy of homography flow data prediction for monocular image sequences.

[0158] In some alternative implementations, the method also includes:

[0159] Spatial detection data is generated by using a homography flow estimation model based on feature data.

[0160] Spatial detection data is used to characterize the three-dimensional spatial features of objects in the first and second images. Typically, spatial detection data can be generated by a spatial detection network included in the model to be trained. Since feature data is a collection of feature maps at multiple scales, feature maps of a certain scale (e.g., the smallest scale) can be selected from these feature maps and input into the spatial detection network to obtain spatial detection data.

[0161] The method provided in this embodiment allows the homography flow estimation model to output spatial detection data, thereby enriching the functionality of the homography flow estimation model and greatly expanding the applicable scenarios of the model.

[0162] In some alternative implementations, the electronic device can generate spatial detection data by following these steps:

[0163] First, based on the feature data, the homography flow estimation model is used to predict the pose between frames, resulting in first spatial detection data representing the relative pose of the camera between the first and second images.

[0164] Then, based on the feature data, the ground equation is predicted by the homography flow estimation model to obtain the second spatial detection data representing the ground equation corresponding to the second sample image.

[0165] The first spatial detection data, i.e., the inter-frame pose prediction data, can be obtained by the inter-frame pose network included in the homography flow estimation model using feature data. The inter-frame pose network can perform calculations on the feature data based on its included network parameters to obtain the inter-frame pose data. It should be noted that existing techniques can be used to calculate inter-frame pose using neural networks, and these will not be elaborated upon here.

[0166] The second spatial detection data, namely ground equation data, can be obtained by the ground equation prediction network included in the homography flow estimation model using feature data. Ground equation data is used to characterize the plane equation within the space where the ground region in the second sample image is located. Typically, ground equation data can include ground normal vectors; the ground equation can be obtained from these ground normal vectors using point normal form, normal form, or other representations.

[0167] The ground equation prediction network can determine the positions of the points in three-dimensional space corresponding to the pixels of the ground region in the second image based on the feature data. Based on the positions of these three-dimensional points, ground equation data including the ground normal vector can be obtained according to existing methods for determining ground normal vectors.

[0168] Typically, the aforementioned inter-frame pose data can be used for vehicle speed estimation or image alignment. The ground equation can be used for depth estimation to obtain camera height information, or to determine the depth information of the image.

[0169] The homography flow estimation model in this embodiment can perform inter-frame pose prediction and ground equation prediction on the input image during prediction using an inter-frame pose network and a ground equation prediction network, which enriches the functionality of the homography flow estimation model and expands its application scenarios.

[0170] Exemplary device

[0171] Figure 9 This is a schematic diagram of the structure of a training apparatus for a homography flow estimation model provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 9As shown, the training device for the homography flow estimation model includes: a first extraction module 901, used to extract features from a first sample image and a second sample image in the same image sequence using the model to be trained, to obtain feature data, wherein the second sample image is located after the first sample image; a first prediction module 902, used to predict based on the feature data and using the model to be trained, to obtain predicted homography flow data representing the relative positional change of the ground region in the first sample image and the second sample image; a first determination module 903, used to determine the homography flow loss value based on the predicted homography flow data and a preset homography flow loss function; and a training module 904, used to adjust the parameters of the model to be trained based on the homography flow loss value until a preset training termination condition is met, to obtain the homography flow estimation model.

[0172] In this embodiment, the first extraction module 901 can use the model to be trained to extract features from the first sample image and the second sample image in the same image sequence to obtain feature data. The second sample image is located after the first sample image. For example, if the first sample image was taken at time t, then the second sample image following it can be taken after time t.

[0173] Typically, the above image sequence is captured by a monocular camera, and the above image sequence can be captured by the camera while it is in motion.

[0174] The model to be trained can include deep neural networks of various structures. For example, the model to be trained can include networks such as ResNet and VarGNet. These networks can be used to extract features from the input image. During feature extraction, the first sample image and the second sample image can be merged on the three RGB channels respectively, and then feature data can be extracted from the merged image.

[0175] In this embodiment, the first prediction module 902 can make predictions based on feature data and using the model to be trained to obtain predicted homography stream data representing the relative positional changes of ground areas in the first sample image and the second sample image.

[0176] Homography refers to the projective mapping of a pixel from one plane to another. Homography stream data is a type of optical flow that represents the instantaneous velocity of a corresponding pixel on the imaging plane of a spatially moving object; that is, the relative displacement feature of a pixel A in one image to a pixel A' in another image. It should be noted that the moving object mentioned above is an object that is moving relative to the imaging plane.

[0177] In this embodiment, the model to be trained may include a homography flow prediction network. The homography flow prediction network can determine the ground regions in the first sample image and the second sample image respectively based on the fused feature data, and then use existing homography estimation methods to determine the mapping relationship between the two ground regions, thereby obtaining homography flow prediction data.

[0178] In this embodiment, the first determining module 903 can determine the homography flow loss value based on the predicted homography flow data and a preset homography flow loss function. The homography flow loss function is used to quantitatively calculate the error between the predicted homography flow data and the actual homography flow data; the homography flow loss value represents the error between the predicted homography flow data and the actual homography flow data.

[0179] In this embodiment, the training module 904 can adjust the parameters of the model to be trained based on the homography flow loss value until the preset training termination condition is met, thereby obtaining the homography flow estimation model.

[0180] The model can be trained using machine learning methods. The preset training termination conditions may include, but are not limited to, at least one of the following: training time exceeds a preset duration; the number of training iterations exceeds a preset number; the homography flow loss value converges.

[0181] Reference Figure 10 , Figure 10 This is a schematic diagram of the structure of a training apparatus for a homography flow estimation model provided in another exemplary embodiment of this disclosure.

[0182] In some alternative implementations, the device further includes: a first detection module 905, used to perform spatial detection using the model to be trained based on feature data, and generate spatial detection data; a second determination module 906, used to determine the spatial detection loss value based on the spatial detection data and a preset spatial detection loss function; and a training module further used to: adjust the parameters of the model to be trained based on the homography flow loss value and the spatial detection loss value.

[0183] In some optional implementations, the spatial detection data includes first spatial detection data and second spatial detection data; the first detection module 905 includes: a first detection unit 9051, used to perform inter-frame pose prediction based on feature data by the model to be trained, to obtain first spatial detection data representing the relative pose of the camera between the first sample image and the second sample image; and a second detection unit 9052, used to perform ground equation prediction based on feature data by the model to be trained, to obtain second spatial detection data representing the ground equation corresponding to the second sample image.

[0184] In some optional implementations, the spatial detection loss function includes a supervised inter-frame pose loss function and a supervised ground equation loss function, and the spatial detection loss value includes a first spatial detection loss value and a second spatial detection loss value; the second determining module 906 includes: a first determining unit 9061, used to determine the first spatial detection loss value based on the first spatial detection data and the corresponding actual inter-frame pose data using the supervised inter-frame pose loss function; and a second determining unit 9062, used to determine the second spatial detection loss value based on the second spatial detection data and the corresponding actual ground equation data using the supervised ground equation loss function.

[0185] In some optional implementations, the homography flow loss function includes a homography flow self-supervised loss function, and the homography flow loss value includes a third homography flow loss value; the first determining module 903 includes: a third determining unit 9031, used to determine the rotation matrix and translation matrix of the second sample image relative to the first sample image based on the first spatial detection data; a fourth determining unit 9032, used to determine the ground normal vector and the height of the camera used to capture the image sequence based on the second spatial detection data; a fifth determining unit 9033, used to determine the homography matrix based on the ground normal vector, preset camera intrinsic parameters, height, rotation matrix, and translation matrix; a sixth determining unit 9034, used to determine the actual homography flow data of the second sample image relative to the first sample image using the homography matrix; and a seventh determining unit 9035, used to determine the third homography flow loss value based on the predicted homography flow data and the actual homography flow data, using the homography flow self-supervised loss function.

[0186] In some optional implementations, the first detection module 905 includes: a tenth determining unit 9053, used to determine a feature map of the target scale from the feature data; and a generating unit 9054, used to perform spatial detection using the model to be trained based on the feature map of the target scale, and generate spatial detection data.

[0187] In some optional implementations, the homography flow loss function includes the homography flow unsupervised loss function, and the homography flow loss value includes the first homography flow loss value; the first determining module 903 includes: an eighth determining unit 9036, used to determine the ground mask data corresponding to the second sample image; a mapping unit 9037, used to map the first sample image to the image plane of the second sample image based on the predicted homography flow data, to obtain a mapped image; and a ninth determining unit 9038, used to determine the first homography flow loss value based on the mapped image, the second sample image, and the ground mask data, using the homography flow unsupervised loss function.

[0188] In some optional implementations, the homography flow loss function includes a homography flow supervision loss function, and the homography flow loss value includes a second homography flow loss value; the first determining module 903 is further configured to: determine the second homography flow loss value based on the actual homography flow data and predicted homography flow data corresponding to the second sample image, using the homography flow supervision loss function.

[0189] The training apparatus for the homography flow estimation model provided in the above embodiments of this disclosure extracts features from a first sample image and a second sample image using the model to be trained. Based on the feature data, it obtains predicted homography flow data of the relative positional changes of the ground region. Then, based on the predicted homography flow data and the homography flow loss function, it determines the homography flow loss value. Finally, based on the homography flow loss value, it adjusts the parameters of the model to be trained to obtain the homography flow estimation model. This achieves the training of the homography flow estimation model. The trained homography flow estimation model can predict homography flow data based on the ground in image sequences captured by a monocular camera. Compared to homography estimation based on the entire image, the homography flow estimation model provided in this disclosure only predicts homography flow for the ground, making the prediction process more targeted and greatly improving the efficiency and accuracy of homography flow data prediction.

[0190] Figure 11 This is a schematic diagram of a homography flow estimation device based on a monocular image provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as… Figure 11 As shown, the homography flow estimation device based on monocular images includes: a second extraction module 1101, used to extract features from a first image and a second image in the same image sequence using a pre-trained homography flow estimation model to obtain feature data, wherein the second image is located after the first image; and a second prediction module 1102, used to predict homography flow based on the feature data using the homography flow estimation model to generate homography flow data representing the relative positional changes of ground regions in the first image and the second image.

[0191] In this embodiment, the second extraction module 1101 can use a pre-trained homography flow estimation model to extract features from the first and second images in the same image sequence to obtain feature data.

[0192] Among them, the homography flow estimation model is based on the above. Figure 2 The image is obtained by training according to the method described in the corresponding embodiment. The second image is located after the first image. For example, if the second sample image was taken at the current time (e.g., time t), then the first image could have been taken before the current time.

[0193] The homography flow estimation model described above can include any suitable deep neural network structure. For example, the model to be trained can include networks such as ResNet (Residual Network) and VarGNet (Variable Group Network), which can be used to extract features from the input image. During feature extraction, the first and second images can be merged on the RGB channels respectively, and then feature data can be extracted from the merged image.

[0194] In this embodiment, the second prediction module 1102 can perform homography flow prediction based on feature data using a homography flow estimation model to generate homography flow data representing the relative positional changes of ground regions in the first and second images.

[0195] Typically, before performing homography flow prediction, feature data can be fused at multiple scales to obtain fused feature data. This fused feature data is then input into the homography flow prediction network. The homography flow prediction network can determine the ground regions in the first and second images based on the fused feature data. Then, using existing homography estimation methods, the mapping relationship between the two ground regions is determined, thus obtaining the homography flow prediction data.

[0196] As an example, feature data can be obtained by downsampling five times from a deep neural network with a Unet structure, then upsampling it several times by a factor of 2 before fusing it with larger-scale features, and so on, to obtain fused feature data at multiple scales. Typically, the fused feature data used in homography flow prediction can be selected from fused feature data at multiple scales. For example, fused feature data at 1 / 2 or 1 / 4 the size of the original image can be selected for homography flow prediction; using fused feature data at these scales results in good model prediction speed and performance.

[0197] It should be understood that the homography flow estimation model used in this embodiment is different from that described above. Figure 2 The homography flow estimation model trained in the corresponding embodiments is the same. For a description of the structure and prediction process of the homography flow estimation model, please refer to the above. Figure 2 The content described in the corresponding embodiments will not be repeated here.

[0198] Reference Figure 12 , Figure 12 This is a schematic diagram of the structure of a homography flow estimation device based on a monocular image provided in another exemplary embodiment of this disclosure.

[0199] In some alternative implementations, the device further includes a second detection module 1103 for performing spatial detection based on feature data using a homography flow estimation model to generate spatial detection data.

[0200] In some optional implementations, the second detection module 1103 includes: a first prediction unit 11031, used to perform inter-frame pose prediction based on feature data using a homography flow estimation model to obtain first spatial detection data representing the camera relative pose between the first image and the second image; and a second prediction unit 11032, used to perform ground equation prediction based on feature data using a homography flow estimation model to obtain second spatial detection data representing the ground equation corresponding to the second sample image.

[0201] The monocular image-based homography flow estimation device provided in the above embodiments of this disclosure extracts features from the first and second images in the same image sequence using a pre-trained homography flow estimation model to obtain feature data. Then, based on the feature data, the homography flow estimation model performs homography flow prediction to generate homography flow data representing the relative positional changes of ground regions in the first and second images. Since the model training uses image sequences captured by a monocular camera to predict homography flow data based on the ground, the prediction process of the model is more targeted, thereby greatly improving the efficiency and accuracy of homography flow data prediction for monocular image sequences.

[0202] Exemplary electronic devices

[0203] Below, for reference Figure 13 To describe an electronic device according to embodiments of the present disclosure. The electronic device may be as follows: Figure 1 The terminal device 101 and server 103 shown, or either one or both, or a standalone device independent of them, can communicate with the terminal device 101 and server 103 to receive the collected input signals from them.

[0204] Figure 13 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0205] like Figure 13 As shown, the electronic device 1300 includes one or more processors 1301 and memory 1302.

[0206] The processor 1301 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1300 to perform desired functions.

[0207] The memory 1302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1301 may execute the program instructions to implement the training method of the homography flow estimation model of the various embodiments of this disclosure described above, or the homography flow estimation method based on monocular images, and / or other desired functions. Various contents such as images and homography flow estimation models may also be stored in the computer-readable storage medium.

[0208] In one example, the electronic device 1300 may also include an input device 1303 and an output device 1304, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0209] For example, when the electronic device is a terminal device 101 or a server 103, the input device 1303 can be a camera, mouse, keyboard, or other similar device, used to input images, execute commands of the above methods, etc. When the electronic device is a standalone device, the input device 1303 can be a communication network connector, used to receive input images from the terminal device 101 and the server 103, execute commands of the above methods, etc.

[0210] The output device 1304 can output various information to the outside, including homography stream data. The output device 1304 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0211] Of course, for the sake of simplicity, Figure 13 Only some of the components of the electronic device 1300 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1300 may include any other suitable components depending on the specific application.

[0212] Exemplary computer program products and computer-readable storage media

[0213] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the training methods for homography flow estimation models according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0214] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0215] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform steps in the training method for a homography flow estimation model according to various embodiments of this disclosure or the homography flow estimation method based on monocular images, as described in the "Exemplary Methods" section of this specification.

[0216] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0217] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0218] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0219] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0220] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0221] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0222] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0223] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A training method for a homography flow estimation model, comprising: The model to be trained is used to extract features from the first sample image and the second sample image in the same image sequence to obtain feature data, wherein the second sample image is located after the first sample image; Based on the feature data, prediction is performed using the model to be trained to obtain predicted homography stream data representing the relative positional changes of ground regions in the first sample image and the second sample image; Based on the predicted homography flow data and the preset homography flow loss function, the homography flow loss value is determined; Based on the homography flow loss value, the parameters of the model to be trained are adjusted until the preset training termination condition is met, and a homography flow estimation model is obtained. The method further includes: Based on the feature data, spatial detection is performed using the model to be trained to generate spatial detection data, which includes at least one of the following: inter-frame pose data of the first sample image and the second sample image, and ground equation data; Based on the spatial detection data and the preset spatial detection loss function, the spatial detection loss value is determined; The step of adjusting the parameters of the model to be trained based on the homography flow loss value includes: The parameters of the model to be trained are adjusted based on the homography flow loss value and the spatial detection loss value.

2. The method according to claim 1, wherein, The spatial detection data includes first spatial detection data and second spatial detection data. The step of generating spatial detection data based on the feature data and using the model to be trained includes: Based on the feature data, the model to be trained performs inter-frame pose prediction to obtain first spatial detection data representing the camera relative pose between the first sample image and the second sample image. Based on the feature data, the ground equation is predicted by the model to be trained, and second spatial detection data representing the ground equation corresponding to the second sample image is obtained.

3. The method according to claim 2, wherein, The spatial detection loss function includes a supervised inter-frame pose loss function and a supervised ground equation loss function, and the spatial detection loss value includes a first spatial detection loss value and a second spatial detection loss value; The step of determining the spatial detection loss value based on the spatial detection data and a preset spatial detection loss function includes: Based on the first spatial detection data and the corresponding actual inter-frame pose data, the first spatial detection loss value is determined using the supervised inter-frame pose loss function. Based on the second spatial detection data and the corresponding actual ground equation data, the second spatial detection loss value is determined using the supervised ground equation loss function.

4. The method according to claim 2, wherein, The homography flow loss function includes the homography flow self-supervised loss function, and the homography flow loss value includes the third homography flow loss value; The step of determining the homography flow loss value based on the predicted homography flow data and a preset homography flow loss function includes: Based on the first spatial detection data, determine the rotation matrix and translation matrix of the second sample image relative to the first sample image; Based on the second spatial detection data, the ground normal vector and the height of the camera used to capture the image sequence are determined; Based on the ground normal vector, preset camera intrinsic parameters, the height, the rotation matrix, and the translation matrix, the homography matrix is ​​determined; Using the homography matrix, the actual homography stream data of the second sample image relative to the first sample image is determined; Based on the predicted homography stream data and the actual homography stream data, the third homography stream loss value is determined using the homography stream self-supervised loss function.

5. The method according to claim 1, wherein, The step of generating spatial detection data by using the model to be trained based on the feature data includes: Determine a feature map of the target scale from the feature data; Based on the feature map at the target scale, the model to be trained performs spatial detection to generate the spatial detection data.

6. The method according to claim 1, wherein, The homography flow loss function includes the homography flow unsupervised loss function, and the homography flow loss value includes the first homography flow loss value; The step of determining the homography flow loss value based on the predicted homography flow data and a preset homography flow loss function includes: Determine the ground mask data corresponding to the second sample image; Based on the predicted homography stream data, the first sample image is mapped onto the image plane of the second sample image to obtain a mapped image; Based on the mapped image, the second sample image, and the ground mask data, the first homography flow loss value is determined using the homography flow unsupervised loss function.

7. The method according to claim 1, wherein, The homography flow loss function includes a homography flow supervision loss function, and the homography flow loss value includes a second homography flow loss value; The step of determining the homography flow loss value based on the predicted homography flow data and a preset homography flow loss function includes: Based on the actual homography flow data corresponding to the second sample image and the predicted homography flow data, the second homography flow loss value is determined using the homography flow supervision loss function.

8. A homography flow estimation method based on monocular images, comprising: Using a pre-trained homography flow estimation model, feature data is obtained by extracting features from the first and second images in the same image sequence, wherein the second image is located after the first image. Based on the feature data, the homography flow estimation model is used to predict homography flow and generate homography flow data representing the relative positional changes of ground areas in the first image and the second image. The pre-trained homography flow estimation model is obtained by training the homography flow estimation model according to any one of claims 1-7.

9. A training device for a homography flow estimation model, comprising: The first extraction module is used to extract features from the first sample image and the second sample image in the same image sequence using the model to be trained, and obtain feature data, wherein the second sample image is located after the first sample image; The first prediction module is used to make predictions based on the feature data and the model to be trained, to obtain predicted homography stream data representing the relative positional changes of ground areas in the first sample image and the second sample image. The first determining module is used to determine the homography flow loss value based on the predicted homography flow data and a preset homography flow loss function; The training module is used to adjust the parameters of the model to be trained based on the homography flow loss value until the preset training termination condition is met, so as to obtain the homography flow estimation model. The device further includes: The first detection module is used to perform spatial detection based on the feature data and the model to be trained, and generate spatial detection data. The spatial detection data includes at least one of the following: inter-frame pose data of the first sample image and the second sample image, and ground equation data. The second determining module is used to determine the spatial detection loss value based on the spatial detection data and the preset spatial detection loss function; The training module is used to adjust the parameters of the model to be trained based on the homography flow loss value and the spatial detection loss value.

10. A homography flow estimation device based on monocular images, comprising: The second extraction module is used to extract features from the first image and the second image in the same image sequence using a pre-trained homography flow estimation model to obtain feature data, wherein the second image is located after the first image; The second prediction module is used to perform homography flow prediction based on the feature data using the homography flow estimation model, and generate homography flow data representing the relative positional changes of ground areas in the first image and the second image; The pre-trained homography flow estimation model is obtained by training the homography flow estimation model according to any one of claims 1-7.

11. A computer-readable storage medium storing a computer program for performing the method according to any one of claims 1-8.

12. An electronic device, the electronic device comprising: processor; Memory for storing the executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-8.

Images