Shooting device pose determination method, shooting device positioning model training method, vehicle control method, computing device, storage medium and computer program product

Abstract

The embodiment of the invention provides a shooting device pose determination method, a shooting device positioning model training method, a vehicle control method, a computing device, a storage medium and a computer program product. The shooting device pose determination method comprises the steps of determining a target image shot by a target shooting device; the target image is input into a shooting equipment positioning model, the equipment pose of the target shooting equipment is obtained, the shooting equipment positioning model is obtained through training according to sample images of different visual angles and rendering images associated with the sample images, and the equipment pose of the target shooting equipment is obtained; the rendering image is obtained by processing the pose of the label device associated with the sample image according to a pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and the reasoning precision, further enables the equipment pose obtained by the shooting equipment positioning model according to the given target image to be more accurate in the model application process, guarantees that the pose estimation process is simple and convenient, and reduces the reasoning cost.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a method for determining the posture of a camera device, a method for training a positioning model for a camera device, a vehicle control method, a computing device, a storage medium, and a computer program product. Background Art

[0002] The pose estimation of the shooting device can be understood as determining the position and orientation of the shooting device in the world coordinate system. The pose estimation of the shooting device can be applied to a variety of tasks, such as the field of scene three-dimensional reconstruction, visual navigation, motion analysis, augmented reality and remote sensing monitoring. Specifically, in the field of visual navigation, autonomous navigation can be performed based on the pose estimation of the shooting device.

[0003] Currently, the pose estimation of a camera can typically be performed by estimating the camera's 3D position and 3D angular deflection in the world coordinate system based on a given image. However, current pose estimation methods are complex, inferentially expensive, and exhibit low accuracy. Consequently, an effective technical solution is urgently needed to address these issues. Summary of the Invention

[0004] In view of this, embodiments of this specification provide two methods for determining the pose of a camera. One or more embodiments of this specification also relate to two apparatuses for determining the pose of a camera, a method for training a camera positioning model, an apparatus for training a camera positioning model, a vehicle control method, a vehicle control device, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a method for determining a posture of a shooting device is provided, comprising:

[0006] Determine a target image captured by a target shooting device;

[0007] The target image is input into a shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on a posture feature extraction model.

[0008] According to a second aspect of the embodiments of this specification, there is provided an apparatus for determining a posture of a photographing device, comprising:

[0009] A determination module configured to determine a target image captured by a target capturing device;

[0010] The input module is configured to input the target image into the shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on the posture feature extraction model.

[0011] According to a third aspect of the embodiments of this specification, a method for determining a posture of a shooting device is provided, which is applied to a cloud-side device, including:

[0012] Receiving a target image sent by a target shooting device;

[0013] The target image is input into a shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on a posture feature extraction model.

[0014] According to a fourth aspect of the embodiments of this specification, a device for determining a posture of a shooting device is provided, which is applied to a cloud-side device, including:

[0015] A receiving module is configured to receive a target image sent by a target shooting device;

[0016] The input module is configured to input the target image into the shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on the posture feature extraction model.

[0017] According to a fifth aspect of the embodiments of this specification, a method for training a camera positioning model is provided, comprising:

[0018] Determine first sample images of different perspectives and tag device poses corresponding to the first sample images, and determine a first rendered image associated with the first sample image, wherein the first rendered image is obtained by processing the tag device pose associated with the sample image according to a pose feature extraction model;

[0019] Inputting the first sample image into a shooting device positioning model to obtain first sample image features and a first predicted device pose associated with the first sample image;

[0020] Inputting the first rendered image into the shooting device positioning model to obtain first rendered image features;

[0021] The shooting device positioning model is trained according to the first sample image feature, the first rendered image feature, the first predicted device pose and the label device pose until a shooting device positioning model that meets a first training stop condition is obtained.

[0022] According to a sixth aspect of the embodiments of this specification, a device for training a camera positioning model is provided, comprising:

[0023] a determination module configured to determine first sample images of different perspectives and tag device poses corresponding to the first sample images, and determine a first rendered image associated with the first sample images, wherein the first rendered image is obtained by processing the tag device poses associated with the sample images according to a pose feature extraction model;

[0024] a first input module configured to input the first sample image into a shooting device positioning model to obtain first sample image features and a first predicted device pose associated with the first sample image;

[0025] a second input module configured to input the first rendered image into the shooting device positioning model to obtain first rendered image features;

[0026] A training module is configured to train the shooting device positioning model according to the first sample image features, the first rendered image features, the first predicted device pose and the label device pose until a shooting device positioning model that meets a first training stop condition is obtained.

[0027] According to a seventh aspect of the embodiments of this specification, a vehicle control method is provided, which is applied to a cloud-side device, including:

[0028] receiving a target image sent by a target shooting device, wherein the target shooting device is disposed on a target vehicle;

[0029] Inputting the target image into a shooting device positioning model to obtain a device pose of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the labeled device pose associated with the sample images based on a pose feature extraction model;

[0030] The driving trajectory of the target vehicle is determined according to the device posture of the target shooting device, and the target vehicle is controlled according to the driving trajectory.

[0031] According to an eighth aspect of the embodiments of this specification, a vehicle control device is provided, which is applied to a cloud-side device, including:

[0032] a receiving module configured to receive a target image sent by a target shooting device, wherein the target shooting device is provided on a target vehicle;

[0033] an input module configured to input the target image into a shooting device positioning model to obtain a device pose of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the labeled device pose associated with the sample images based on a pose feature extraction model;

[0034] The control module is configured to determine the driving trajectory of the target vehicle according to the device posture of the target shooting device, and control the target vehicle according to the driving trajectory.

[0035] According to a ninth aspect of the embodiments of this specification, a computing device is provided, including:

[0036] memory and processor;

[0037] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above method are implemented.

[0038] According to a tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and the instructions implement the steps of the above method when executed by a processor.

[0039] According to an eleventh aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0040] An embodiment of the present specification provides a method for determining the posture of a shooting device, which determines a target image captured by a target shooting device; inputs the target image into a shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered image is obtained by processing the label device posture associated with the sample image based on a posture feature extraction model.

[0041] In the above method, when training the shooting device positioning model, a rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of an application scenario of a method for determining the posture of a shooting device provided by an embodiment of this specification;

[0043] FIG2( a ) is a flow chart of a method for determining the posture of a photographing device provided in one embodiment of this specification;

[0044] FIG2( b ) is a schematic diagram of the processing process of the posture feature extraction model in the method for determining the posture of a shooting device provided in one embodiment of this specification;

[0045] Figure 3 This is a flowchart of a supervised training process in a method for determining a camera posture provided by one embodiment of this specification;

[0046] Figure 4 This is a flowchart of the processing process of unsupervised training in a method for determining the posture of a shooting device provided by one embodiment of this specification;

[0047] Figure 5 This is a schematic structural diagram of a device for determining the position and posture of a photographing device provided by one embodiment of this specification;

[0048] Figure 6 is a flowchart of another method for determining the position and posture of a photographing device provided according to one embodiment of this specification;

[0049] Figure 7 This is a schematic structural diagram of another apparatus for determining the position and posture of a photographing device provided by one embodiment of this specification;

[0050] Figure 8 This is a flow chart of a method for training a camera positioning model provided by one embodiment of this specification;

[0051] Figure 9 This is a structural diagram of a shooting device positioning model training device provided by an embodiment of this specification;

[0052] Figure 10 is a flow chart of a vehicle control method provided by one embodiment of this specification;

[0053] Figure 11 This is a schematic structural diagram of a vehicle control device provided by one embodiment of this specification;

[0054] Figure 12 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0055] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0056] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0058] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0059] First, the terms involved in one or more embodiments of this specification are explained.

[0060] Camera pose refers to the position and orientation of a camera in three-dimensional space. It is typically represented by a translation vector (the x, y, and z coordinates describing the camera's position) and a rotation matrix or quaternion (a method describing the camera's orientation). In computer vision and computer graphics, determining camera pose is crucial for reconstructing, tracking, and rendering three-dimensional scenes.

[0061] Camera pose estimation: also known as camera attitude estimation or visual localization, refers to the process of inferring the position and orientation of a camera in three-dimensional space by analyzing image or video data.

[0062] Absolute Pose Regression (APR) refers to the process of directly calculating the position and orientation of a camera in a global coordinate system in three-dimensional space by analyzing image or video data.

[0063] Neural Radiance Fields (NeRF) is a method for generating realistic 3D scenes. NeRF aims to learn a 3D representation of a scene, including its geometry and surface reflectance, from a single or limited number of views. NeRF trains a neural network to represent the radiance and color of each 3D point in a scene. By capturing images of the scene from different orientations and combining them with camera parameters, a neural network model can be trained to render the scene from new perspectives.

[0064] Volume Rendering: Volume rendering is a technique for visualizing three-dimensional volume data. It is widely used in fields such as medical image processing, scientific visualization, and computer graphics. Volume data is typically composed of densely packed three-dimensional pixels (or voxels), each containing information about material density, color, or other properties. The goal of volume rendering is to convert this voxel data into visual images that enable better understanding and analysis of internal structures and features.

[0065] Structure from Motion (SfM): Structure from Motion is a computer vision technique used to recover the geometric structure and camera motion of a 3D scene from an image sequence. An SfM model refers to a 3D scene model generated using SfM.

[0066] MLP (Multilayer Perceptron) is a feedforward neural network and a deep learning model. An MLP consists of multiple layers of neurons, each containing several nodes connected by nonlinear activation functions to form a hierarchical structure. In an MLP, information propagates from the input layer forward through a series of hidden layers before reaching the output layer. Except for the input layer, each layer of neurons performs a weighted summation of the signals it receives and generates an output using a nonlinear activation function. This output then serves as the input for the next layer of neurons. This layer-by-layer processing captures complex, nonlinear relationships in the input data.

[0067] This specification provides two methods for determining the pose of a camera. One or more embodiments of this specification also involve two apparatuses for determining the pose of a camera, a method for training a camera positioning model, an apparatus for training a camera positioning model, a vehicle control method, a vehicle control apparatus, a computing device, a computer-readable storage medium, and a computer program product, each of which is described in detail in the following embodiments.

[0068] See also Figure 1 , Figure 1 A schematic diagram of an application scenario of a method for determining the posture of a shooting device provided according to an embodiment of this specification is shown.

[0069] Figure 1 The system includes a terminal device 102 and a cloud device 104. The terminal device 102 may be, for example, a camera, a video camera, a driving recorder or other shooting device.

[0070] In specific implementations, in the field of autonomous driving, the end-side device 102 can be deployed on a target vehicle to capture road conditions in front of the target vehicle. The end-side device 102 can send the captured target image to the cloud-side device 104. The cloud-side device 104 can input the target image into the positioning model of the capture device to obtain the device pose of the end-side device 102. Subsequently, the device pose can be used to control the driving of the target vehicle.

[0071] When training the camera positioning model, training can be performed based on sample images from different perspectives and rendered images associated with the sample images. Specifically, supervised training and unsupervised training can be performed separately. A pose feature extraction model is introduced, and the pose feature extraction model is used to determine the rendered image associated with the sample image, thereby ensuring the robustness and accuracy of the camera positioning model training. Specifically, the rendered image associated with the sample image can be obtained using the pose feature extraction model. Specifically, the pose of the label device associated with the sample image can be input into the pose feature extraction model. In the pose feature extraction model, the pose of the label device is processed to obtain image parameters of the sample image, and the image parameters are subjected to volume rendering processing to obtain the rendered image associated with the sample image.

[0072] The cloud-side device 104 can be understood as a server that provides various services, including physical servers and cloud servers. For example, a server that provides communication services to multiple clients, a server that provides background training to support models used on clients, and a server that processes data sent by clients. It should be noted that the cloud-side device 104 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The cloud-side device 104 can also be a server for a distributed system, or a server that integrates a blockchain. The cloud-side device 104 can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0073] It is worth noting that the method for determining the posture of the shooting device provided in the embodiments of this specification can be executed by the cloud-side device 104. In other embodiments of this specification, the shooting device positioning model can be deployed in the terminal-side device 102, so that the terminal-side device 102 can also have similar functions as the cloud-side device 104, thereby executing the method for determining the posture of the shooting device provided in the embodiments of this specification; in other embodiments, the method for determining the posture of the shooting device provided in the embodiments of this specification can also be executed jointly by the terminal-side device 102 and the cloud-side device 104.

[0074] Referring to FIG. 2( a ), FIG. 2( a ) shows a flow chart of a method for determining a posture of a shooting device provided according to an embodiment of the present specification, which specifically includes the following steps.

[0075] Step 202: Determine a target image captured by a target shooting device.

[0076] Specifically, the camera pose determination method provided in the embodiments of this specification can be applied to fields such as augmented reality, robotic navigation, and autonomous driving. Specifically, after determining the camera pose, the target position captured by the camera can be further calculated to achieve scene perception. Furthermore, the camera, with its pose determined, can be combined with other sensors, such as an inertial measurement unit sensor.

[0077] The target shooting device may include a video camera, a camera, a mobile phone, a driving recorder and other shooting devices.

[0078] During specific implementation, a target image captured by a target capturing device may be received.

[0079] For example, a target image captured by a camera installed on the robot may be received.

[0080] Step 204: Input the target image into the shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on the posture feature extraction model.

[0081] Among them, the shooting device positioning model can be understood as a model for determining the posture of the shooting device. The device posture of the target shooting device can be understood as the position and orientation of the target shooting device in three-dimensional space. The device posture of the target shooting device is the device positioning result for the target shooting device. The sample image can be understood as an image taken by the shooting device, which can be used as a training sample for training the shooting device positioning model. The labeled device posture associated with the sample image can be understood as the actual device posture when the shooting device took the sample image. The labeled device posture can be used as a training label to train the shooting device positioning model together with the training sample.

[0082] In practical applications, the camera positioning model can be a neural network model based on APR.

[0083] Based on this, after receiving the target image captured by the target shooting device, the target image can be input into the shooting device positioning model to obtain the device posture of the target shooting device.

[0084] Continuing with the above example, the target image can be input into the camera positioning model to obtain the position and pose of the camera on the robot, so that the robot movement can be controlled based on the position and pose of the camera.

[0085] In practical applications, the training steps of the camera positioning model include:

[0086] Determine first sample images of different perspectives and tag device poses corresponding to the first sample images, and determine a first rendered image associated with the first sample images;

[0087] Inputting the first sample image into a shooting device positioning model to obtain first sample image features and a first predicted device pose associated with the first sample image;

[0088] Inputting the first rendered image into the shooting device positioning model to obtain first rendered image features;

[0089] The shooting device positioning model is trained according to the first sample image feature, the first rendered image feature, the first predicted device pose and the label device pose until a shooting device positioning model that meets a first training stop condition is obtained.

[0090] Among them, the label device pose corresponding to the first sample image can be understood as the actual device pose of the target shooting device when the first sample image is taken. The first sample image may include multiple, and the shooting angle of each first sample image is different. The first training stop condition can be understood as the model loss value reaching a preset loss value threshold and / or the number of training times reaching a preset number threshold. The first predicted device pose associated with the first sample image can be understood as an absolute pose, and the absolute pose can be understood as the position and direction of the target shooting device that shoots the first sample image in the global coordinate system of the three-dimensional space.

[0091] Based on this, it is possible to determine multiple first sample images captured by the target shooting device at different shooting angles, as well as the label device posture corresponding to each first sample image, and determine the first rendered image associated with each first sample image. Then, one of the multiple first sample images, the label device posture corresponding to the first sample image, and the first rendered image associated with the first sample image constitute a training sample group. It will be understood that the embodiments of this specification are explained in detail using the example of training the shooting device positioning model according to a training sample group, and the process of training the shooting device positioning model according to other training sample groups is similar.

[0092] Then, when training the shooting device positioning model based on this training sample group, the first sample image can be input into the shooting device positioning model to obtain the first sample image features of the first sample image and the first predicted device pose associated with the first sample image, and the first rendered image can be input into the shooting device positioning model to obtain the first rendered image features of the first rendered image. The shooting device positioning model can be trained based on the first sample image features, the first rendered image features, the first predicted device pose and the labeled device pose until a shooting device positioning model that meets the first training stop condition is obtained.

[0093] In practical applications, the first sample image can be further enhanced using an online random view synthesis data augmentation strategy (i.e., RVS). Furthermore, during training, the image feature triplet error (i.e., subsequent pose feature error, pose error, and image error) can be used to further mitigate image domain differences and prevent feature collapse.

[0094] In summary, by training the camera positioning model on a labeled training dataset, the reasoning performance of the camera positioning model is guaranteed.

[0095] In a specific implementation, determining the first rendered image associated with the first sample image includes:

[0096] Inputting the tag device posture associated with the first sample image into the posture feature extraction model;

[0097] In the posture feature extraction model, the posture of the tag device is processed to obtain image parameters of the first sample image;

[0098] Perform volume rendering processing on the image parameters to obtain a first rendered image associated with the first sample image.

[0099] The posture feature extraction model can be understood as a neural network model based on a neural radiation field. The posture feature extraction model can extract neural posture features of the device posture of the target shooting device. The image parameters of the first sample image can include an opacity parameter (i.e., a σ parameter) and a color parameter (i.e., an RGB parameter) of the first sample image.

[0100] In a specific implementation, the image parameters include opacity parameters and color parameters;

[0101] The performing volume rendering processing on the image parameters to obtain a first rendered image associated with the first sample image includes:

[0102] Volume rendering is performed on the opacity parameter and the color parameter to obtain a first rendered image associated with the first sample image.

[0103] In addition, after inputting the tag device posture associated with the first sample image into the posture feature extraction model, the method further includes:

[0104] In the posture feature extraction model, processing the posture of the tag device to obtain a posture embedding vector associated with the first sample image;

[0105] performing volume rendering on the opacity parameter and the pose embedding vector to obtain a first predicted device pose feature associated with the first sample image;

[0106] Determine a second predicted device pose associated with the first sample image according to the first predicted device pose feature.

[0107] The first predicted device posture feature associated with the first sample image is the neural posture feature.

[0108] Specifically, referring to FIG2(b), FIG2(b) shows a schematic diagram of the processing process of the pose feature extraction model in the method for determining the pose of a shooting device provided according to an embodiment of the present specification. As shown in FIG2(b), the pose feature extraction model may include an MLP model and a pose embedding branch. Specifically, the label device pose (i.e., the actual device pose) of the first sample image may be input into the pose feature extraction model, and the MLP model is used to extract the predicted device pose to obtain a series of incident and outgoing light rays of the shooting device. Each light ray passes through a series of points in 3D space. The position and light direction of each point are obtained through MLP regression. The position and light direction of each point are processed using the pose branch to obtain the RGB parameters, σ parameters, and pose embedding vector of the first sample image. Based on the σ parameters and the pose embedding vector, a first predicted device pose feature (i.e., a feature map with pose information) associated with the first sample image is obtained through volume rendering processing. At the same time, based on the RGB parameters and σ parameters, a first rendered image is obtained through volume rendering processing. The first rendered image and the first predicted device pose feature can be used for subsequent training of the pose feature extraction model. In addition, after obtaining the first predicted device posture feature, the first predicted device posture feature can be input into the decoder to obtain the predicted device posture output by the decoder, and the posture feature extraction model is trained based on the output predicted device posture and the input labeled device posture.

[0109] In another embodiment of the present specification, the neural radiation field can be replaced by other rendering modules that can introduce posture features. Introducing a better new perspective rendering module can improve the model training speed and reasoning accuracy.

[0110] In summary, by introducing the correlation features of absolute pose and pose (i.e., neural pose features), the accuracy and robustness of the absolute pose regression method are guaranteed. The funny reasoning of the absolute pose regressor is used to overcome the problems of cumbersome process and high inference overhead, and a balance is achieved among the overhead, speed, and accuracy of camera positioning.

[0111] In one embodiment of this specification, after determining the second predicted device pose associated with the first sample image, the method further includes:

[0112] The posture feature extraction model is trained according to the label device posture and the second predicted device posture.

[0113] Specifically, this training phase can be understood as the neural rendering phase for the pose feature extraction model. By adding a pose embedding branch to the pose feature extraction model and obtaining a first predicted device pose feature based on volume rendering, in order to ensure that the first predicted device pose feature contains the original camera pose information, the model can be trained to supervise its ability to restore the labeled device pose through the decoder. It can be understood that this training phase is the first phase, used to train the pose feature extraction model so that the pose feature extraction model learns new perspective rendering and camera pose feature extraction.

[0114] Based on this, the model loss value can be calculated according to the label device posture and the first predicted device posture, and the posture feature extraction model can be trained according to the model loss value.

[0115] In summary, through the first stage of training, the pose feature extraction model learns new perspective rendering and shooting device pose feature extraction.

[0116] In addition, after performing volume rendering on the opacity parameter and the pose embedding vector to obtain a first predicted device pose feature associated with the first sample image, the method further includes:

[0117] Inputting the first predicted device posture into the posture feature extraction model to obtain a second predicted device posture feature associated with the first sample image;

[0118] The posture feature extraction model is trained according to the first predicted device posture feature and the second predicted device posture feature.

[0119] This training can be understood as the second phase of training the pose feature extraction model. The second phase, as the absolute pose estimation phase, is primarily used to train the camera positioning model. Specifically, the pose feature extraction model can be used to train the camera positioning model (i.e., APRnet) on a labeled dataset.

[0120] Specifically, since the first rendered image used to train the camera positioning model is determined based on the pose feature extraction model, the pose feature extraction model can also be trained to indirectly improve the model performance of the camera positioning model. A first predicted device pose corresponding to the first sample image is input into the pose feature extraction model to obtain a second predicted device pose feature associated with the first sample image. A pose feature error is calculated based on the first and second predicted device pose features, and the pose feature extraction model is trained based on this pose feature error.

[0121] In summary, by training the pose feature extraction model based on pose feature errors, the feature extraction performance of the pose feature extraction model is further guaranteed, thereby ensuring the quality of the first rendered image and the performance of the camera positioning model trained based on the first rendered image. Furthermore, the effectiveness of the neural pose feature introduction can improve the accuracy and robustness of the absolute pose regression method.

[0122] In a specific implementation, the training of the shooting device positioning model according to the first sample image feature, the first rendered image feature, the first predicted device pose, and the label device pose until a shooting device positioning model that satisfies a first training stop condition is obtained includes:

[0123] Calculating a first model loss value according to the first sample image feature and the first rendered image feature;

[0124] Calculating a second model loss value according to the first predicted device pose and the tag device pose;

[0125] The shooting device positioning model is trained according to the first model loss value and the second model loss value until a shooting device positioning model that meets a first training stop condition is obtained.

[0126] The first model loss value can be understood as the image error between the first sample image feature and the first rendered image feature, and the second model loss value can be understood as the pose error between the predicted device pose and the labeled device pose.

[0127] Specifically, the image error can be calculated as the first model loss value based on the first sample image features and the first rendered image features, and the posture error can be calculated as the second model loss value based on the predicted device posture and the labeled device posture. The shooting device positioning model is trained according to the first model loss value and the second model loss value until a shooting device positioning model that meets the first training stop condition is obtained.

[0128] In practical applications, the MSE error can be calculated based on the first predicted device posture and the label device posture, and the cosine similarity error can be calculated for the first predicted device posture feature and the second predicted device posture feature.

[0129] In summary, by introducing multiple loss functions to train the camera positioning model, the reasoning performance of the camera positioning model is guaranteed.

[0130] Specifically, to further improve model accuracy, a large amount of unlabeled datasets can be used to perform unsupervised refinement of the camera positioning model. The specific implementation method is as follows:

[0131] The method further includes training the shooting device positioning model according to the first sample image feature, the first rendered image feature, the first predicted device pose, and the label device pose until a shooting device positioning model that meets a first training stop condition is obtained:

[0132] determining a second sample image, and determining a second rendered image associated with the second sample image;

[0133] Inputting the second sample image into the shooting device positioning model to obtain second sample image features;

[0134] Inputting the second rendered image into the shooting device positioning model to obtain second rendered image features;

[0135] The shooting device positioning model is trained according to the second sample image feature and the second rendered image feature until a shooting device positioning model that meets a second training stop condition is obtained.

[0136] The second training stop condition can be the same as the first training stop condition. The second sample image can be understood as a sample image without a label (i.e., without a labeled device pose). It is understandable that the second sample image can also include multiple images with different shooting angles.

[0137] Specifically, the second sample image and the second rendered image can be input into the shooting device positioning model to obtain the second sample image features of the second sample image and the second rendered image features of the second rendered image. The image error is calculated based on the second sample image and the second rendered image features, and the shooting device positioning model is trained based on the image error until a shooting device positioning model that meets the second training stop condition is obtained.

[0138] It can be understood that this stage is the third stage of training. As the self-supervised pose refinement stage, the third stage can use the pose feature extraction model to refine the shooting device positioning model on the unlabeled dataset.

[0139] In a specific implementation, determining the second rendered image associated with the second sample image includes:

[0140] Inputting the second sample image into the shooting device positioning model to obtain a third predicted device pose associated with the second sample image;

[0141] The third predicted device posture is input into the posture feature extraction model to obtain a second rendered image associated with the second sample image.

[0142] It can be understood that the process of inputting the third predicted device posture into the posture feature extraction model to obtain the second rendered image associated with the second sample image is similar to the aforementioned process of determining the first rendered image associated with the first sample image, and will not be repeated here.

[0143] In addition, after inputting the second rendered image into the shooting device positioning model, the method further includes:

[0144] obtaining a fourth predicted device pose associated with the second rendered image;

[0145] Accordingly, the method further comprises:

[0146] Inputting the third predicted device posture into the posture feature extraction model to obtain a third predicted device posture feature;

[0147] Inputting the fourth predicted device posture into the posture feature extraction model to obtain a fourth predicted device posture feature;

[0148] The posture feature extraction model is trained according to the third predicted device posture feature and the fourth predicted device posture feature.

[0149] The third predicted device pose can be understood as a predicted pose obtained based on the second sample image, and the fourth predicted device pose can be understood as a predicted pose of a rendered image obtained based on the second rendered image.

[0150] Specifically, the third predicted device posture and the fourth predicted device posture can be input into the posture feature extraction model to obtain the third predicted device posture features and the fourth predicted device posture features, and the posture feature error can be calculated based on the third predicted device posture features and the fourth predicted device posture features, and the posture feature extraction model can be trained based on the posture feature error.

[0151] To sum up, in the above method, when training the shooting device positioning model, the rendered image associated with the sample image is introduced. The training of the shooting device positioning model based on the rendered image can learn the rendering of the new perspective, and the volume rendering is performed through the pose feature extraction model to determine the rendered image associated with the sample image, enriching the training data of the shooting device positioning model. The efficient reasoning of the pose feature extraction model based on the neural radiation field can improve the model training speed and reasoning accuracy. Further, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring the simplicity of the pose estimation process, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0152] The following combined Figure 3 and attached Figure 4Taking the application of the method for determining the position of a camera provided in this specification in the training of a positioning model for a camera as an example, the method for determining the position of a camera is further described. Figure 3 A flowchart of a supervised training process in a method for determining the posture of a shooting device provided in an embodiment of this specification is shown, which specifically includes the following steps.

[0153] Step 302: Determine first sample images of different viewing angles and the tag device poses corresponding to the first sample images.

[0154] Step 304: Input the first sample image into a shooting device positioning model (ie, an APR network model) to obtain first sample image features and a first predicted device pose associated with the first sample image.

[0155] Step 306: Input the label device posture associated with the first sample image into the posture feature extraction model to obtain a first predicted device posture feature associated with the first sample image and a first rendered image associated with the first sample image.

[0156] Step 308: Input the first predicted device posture associated with the first sample image into the posture feature extraction model (ie, the NeRF-P network model) to obtain the second predicted device posture feature associated with the first sample image.

[0157] Step 310: Train the shooting device positioning model according to the first sample image features, the first rendered image features, the first predicted device pose and the label device pose until a shooting device positioning model that meets a first training stop condition is obtained.

[0158] Specifically, the image error can be calculated as the first model loss value based on the first sample image features and the first rendered image features, and the posture error can be calculated as the second model loss value based on the predicted device posture and the labeled device posture. The shooting device positioning model is trained according to the first model loss value and the second model loss value until a shooting device positioning model that meets the first training stop condition is obtained.

[0159] Step 312: Training the posture feature extraction model based on the first predicted device posture feature and the second predicted device posture feature.

[0160] Specifically, the posture feature error can be calculated based on the first predicted device posture feature and the second predicted device posture feature, and the posture feature extraction model can be trained based on the posture feature error.

[0161] Specifically, the process described in steps 302 to 312 is the absolute pose estimation stage, in which the pose feature extraction model can be used to train the camera positioning model on the labeled data set.

[0162] Furthermore, in order to improve the accuracy of the camera positioning model, after the supervised training of the camera positioning model using the absolute pose estimation stage, a self-supervised pose refinement stage can be performed, in which the pose feature extraction model can be used to refine the camera positioning model on an unlabeled dataset. Figure 4 , Figure 4 A flowchart of the processing process of unsupervised training in a method for determining the posture of a shooting device provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0163] Step 402: Determine a second sample image, input the second sample image into the shooting device positioning model, and obtain second sample image features and a third predicted device pose.

[0164] Step 404: Input the third predicted device posture into the posture feature extraction model to obtain a second rendered image associated with the third predicted device posture feature and the second sample image.

[0165] Step 406: Input the second rendered image into the shooting device positioning model to obtain second rendered image features and a fourth predicted device pose.

[0166] Step 408: Input the fourth predicted device posture into the posture feature extraction model to obtain the fourth predicted device posture feature.

[0167] Step 410: training the shooting device positioning model according to the second sample image features and the second rendered image features until a shooting device positioning model that meets a second training stop condition is obtained.

[0168] Step 412: Training the posture feature extraction model based on the third predicted device posture feature and the fourth predicted device posture feature.

[0169] To sum up, in the above method, when training the shooting device positioning model, a rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0170] Corresponding to the above method embodiment, this specification also provides an embodiment of a device for determining the posture of a shooting device, Figure 5 FIG. 1 shows a schematic diagram of a device for determining the posture of a photographing device according to an embodiment of the present invention. Figure 5 As shown, the device includes:

[0171] A determination module 502 is configured to determine a target image captured by a target capturing device;

[0172] The input module 504 is configured to input the target image into the shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on the posture feature extraction model.

[0173] In an optional embodiment, the apparatus further includes a training module configured to:

[0174] Determine first sample images of different perspectives and tag device poses corresponding to the first sample images, and determine a first rendered image associated with the first sample images;

[0175] Inputting the first sample image into a shooting device positioning model to obtain first sample image features and a first predicted device pose associated with the first sample image;

[0176] Inputting the first rendered image into the shooting device positioning model to obtain first rendered image features;

[0177] The shooting device positioning model is trained according to the first sample image feature, the first rendered image feature, the first predicted device pose and the label device pose until a shooting device positioning model that meets a first training stop condition is obtained.

[0178] In an optional embodiment, the training module is further configured to:

[0179] Inputting the tag device posture associated with the first sample image into the posture feature extraction model;

[0180] In the posture feature extraction model, the posture of the tag device is processed to obtain image parameters of the first sample image;

[0181] Perform volume rendering processing on the image parameters to obtain a first rendered image associated with the first sample image.

[0182] In an optional embodiment, the training module is further configured to:

[0183] The image parameters include opacity parameters and color parameters;

[0184] The performing volume rendering processing on the image parameters to obtain a first rendered image associated with the first sample image includes:

[0185] Volume rendering is performed on the opacity parameter and the color parameter to obtain a first rendered image associated with the first sample image.

[0186] In an optional embodiment, the training module is further configured to:

[0187] After inputting the tag device posture associated with the first sample image into the posture feature extraction model, the method further includes:

[0188] In the posture feature extraction model, processing the posture of the tag device to obtain a posture embedding vector associated with the first sample image;

[0189] performing volume rendering on the opacity parameter and the pose embedding vector to obtain a first predicted device pose feature associated with the first sample image;

[0190] Determine a second predicted device pose associated with the first sample image according to the first predicted device pose feature.

[0191] In an optional embodiment, the training module is further configured to:

[0192] The posture feature extraction model is trained according to the label device posture and the second predicted device posture.

[0193] In an optional embodiment, the training module is further configured to:

[0194] Inputting the first predicted device posture into the posture feature extraction model to obtain a second predicted device posture feature associated with the first sample image;

[0195] The posture feature extraction model is trained according to the first predicted device posture feature and the second predicted device posture feature.

[0196] In an optional embodiment, the training module is further configured to:

[0197] Calculating a first model loss value according to the first sample image feature and the first rendered image feature;

[0198] Calculating a second model loss value according to the first predicted device pose and the tag device pose;

[0199] The shooting device positioning model is trained according to the first model loss value and the second model loss value until a shooting device positioning model that meets a first training stop condition is obtained.

[0200] In an optional embodiment, the training module is further configured to:

[0201] determining a second sample image, and determining a second rendered image associated with the second sample image;

[0202] Inputting the second sample image into the shooting device positioning model to obtain second sample image features;

[0203] Inputting the second rendered image into the shooting device positioning model to obtain second rendered image features;

[0204] The shooting device positioning model is trained according to the second sample image feature and the second rendered image feature until a shooting device positioning model that meets a second training stop condition is obtained.

[0205] In an optional embodiment, the training module is further configured to:

[0206] Inputting the second sample image into the shooting device positioning model to obtain a third predicted device pose associated with the second sample image;

[0207] The third predicted device posture is input into the posture feature extraction model to obtain a second rendered image associated with the second sample image.

[0208] In an optional embodiment, the training module is further configured to:

[0209] After inputting the second rendered image into the shooting device positioning model, the method further includes:

[0210] obtaining a fourth predicted device pose associated with the second rendered image;

[0211] Inputting the third predicted device posture into the posture feature extraction model to obtain a third predicted device posture feature;

[0212] Inputting the fourth predicted device posture into the posture feature extraction model to obtain a fourth predicted device posture feature;

[0213] The posture feature extraction model is trained according to the third predicted device posture feature and the fourth predicted device posture feature.

[0214] To sum up, in the above-mentioned device, when training the shooting device positioning model, a rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0215] The above is a schematic diagram of a device for determining the position and posture of a camera according to this embodiment. It should be noted that the technical solution of the device for determining the position and posture of a camera is based on the same concept as the technical solution of the method for determining the position and posture of a camera described above. For details not described in detail in the technical solution of the device for determining the position and posture of a camera, please refer to the description of the technical solution of the method for determining the position and posture of a camera described above.

[0216] See also Figure 6 , Figure 6 A flowchart of another method for determining the posture of a shooting device provided according to an embodiment of this specification is shown, which is applied to a cloud-side device and specifically includes the following steps.

[0217] Step 602: Receive a target image sent by a target shooting device;

[0218] Step 604: Input the target image into the shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on the posture feature extraction model.

[0219] In summary, the above method can be applied to cloud-side devices. The cloud-side devices can estimate the device pose of the target image sent by the target shooting device by calling the shooting device positioning model. Moreover, when training the shooting device positioning model, the rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is made more accurate, thereby ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0220] The above is a schematic scheme of a method for determining the position and posture of a camera device according to this embodiment. It should be noted that the technical scheme of this method for determining the position and posture of a camera device and the technical scheme of the aforementioned method for determining the position and posture of a camera device are based on the same concept. For details not described in detail in the technical scheme of the method for determining the position and posture of a camera device, please refer to the description of the technical scheme of the aforementioned method for determining the position and posture of a camera device.

[0221] Corresponding to the above method embodiment, this specification also provides an embodiment of a device for determining the posture of a shooting device, Figure 7 FIG. 1 shows a schematic diagram of the structure of another apparatus for determining the position and posture of a photographing device provided by an embodiment of this specification. Figure 7 As shown, the device includes:

[0222] The receiving module 702 is configured to receive a target image sent by a target shooting device;

[0223] The input module 704 is configured to input the target image into the shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on the posture feature extraction model.

[0224] To sum up, in the above-mentioned device, when training the shooting device positioning model, a rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0225] The above is a schematic diagram of a device for determining the position and posture of a camera according to this embodiment. It should be noted that the technical solution of the device for determining the position and posture of a camera is based on the same concept as the technical solution of the method for determining the position and posture of a camera described above. For details not described in detail in the technical solution of the device for determining the position and posture of a camera, please refer to the description of the technical solution of the method for determining the position and posture of a camera described above.

[0226] See also Figure 8 , Figure 8 A flowchart of a method for training a camera positioning model according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0227] Step 802: Determine first sample images of different perspectives and tag device poses corresponding to the first sample images, and determine a first rendered image associated with the first sample images, wherein the first rendered image is obtained by processing the tag device poses associated with the sample images according to a pose feature extraction model;

[0228] Step 804: Input the first sample image into a shooting device positioning model to obtain first sample image features and a first predicted device pose associated with the first sample image;

[0229] Step 806: inputting the first rendered image into the shooting device positioning model to obtain first rendered image features;

[0230] Step 808: Train the shooting device positioning model according to the first sample image features, the first rendered image features, the first predicted device pose and the label device pose until a shooting device positioning model that meets the first training stop condition is obtained.

[0231] To sum up, in the above method, when training the shooting device positioning model, a rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0232] The above is a schematic scheme of a method for training a camera positioning model according to this embodiment. It should be noted that the technical scheme of this method for training a camera positioning model is based on the same concept as the technical scheme of the aforementioned method for determining the position and posture of the camera. For details not described in detail in the technical scheme of the training method for training a camera positioning model, please refer to the description of the technical scheme of the aforementioned method for determining the position and posture of the camera.

[0233] Corresponding to the above method embodiment, this specification also provides an embodiment of a shooting device positioning model training device, Figure 9 FIG. 1 shows a schematic diagram of a structure of a shooting device positioning model training device provided by an embodiment of this specification. Figure 9 As shown, the device includes:

[0234] A determination module 902 is configured to determine first sample images of different perspectives and tag device poses corresponding to the first sample images, and determine a first rendered image associated with the first sample images, wherein the first rendered image is obtained by processing the tag device poses associated with the sample images according to a pose feature extraction model;

[0235] A first input module 904 is configured to input the first sample image into a capture device positioning model to obtain first sample image features and a first predicted device pose associated with the first sample image;

[0236] A second input module 906 is configured to input the first rendered image into the shooting device positioning model to obtain first rendered image features;

[0237] The training module 908 is configured to train the shooting device positioning model according to the first sample image features, the first rendered image features, the first predicted device pose and the label device pose until a shooting device positioning model that meets the first training stop condition is obtained.

[0238] To sum up, in the above-mentioned device, when training the shooting device positioning model, a rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0239] The above is a schematic diagram of a camera positioning model training device according to this embodiment. It should be noted that the technical solution of this camera positioning model training device and the technical solution of the aforementioned camera position determination method are based on the same concept. For details not described in detail in the technical solution of the camera positioning model training device, please refer to the description of the technical solution of the aforementioned camera position determination method.

[0240] See also Figure 10 , Figure 10 A flow chart of a vehicle control method provided according to an embodiment of the present specification is shown, which is applied to a cloud-side device and specifically includes the following steps.

[0241] Step 1002: receiving a target image sent by a target shooting device, wherein the target shooting device is set on a target vehicle;

[0242] Step 1004: Input the target image into a camera positioning model to obtain a device pose of the target camera, wherein the camera positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are processed based on the labeled device pose associated with the sample images using a pose feature extraction model.

[0243] Step 1006: Determine the driving trajectory of the target vehicle according to the device posture of the target shooting device, and control the target vehicle according to the driving trajectory.

[0244] Specifically, it is possible to receive a target image sent by a target shooting device installed on a target vehicle, and input the target image into a shooting device positioning model to obtain the device posture of the target shooting device, and determine the current position of the target vehicle and / or the distance between it and the target object in the target image based on the device posture, thereby generating a driving trajectory of the target vehicle and controlling the target vehicle to travel according to the driving trajectory.

[0245] To sum up, in the above method, when training the shooting device positioning model, a rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0246] The above is a schematic diagram of a vehicle control method according to this embodiment. It should be noted that the technical solution of this vehicle control method and the technical solution of the aforementioned method for determining the position and posture of a camera device are based on the same concept. For details not described in detail in the technical solution of the vehicle control method, please refer to the description of the technical solution of the aforementioned method for determining the position and posture of a camera device.

[0247] Corresponding to the above method embodiment, this specification also provides a vehicle control device embodiment, Figure 11 FIG1 shows a schematic diagram of the structure of a vehicle control device provided by an embodiment of this specification. Figure 11 As shown, the device includes:

[0248] The receiving module 1102 is configured to receive a target image sent by a target shooting device, wherein the target shooting device is set on the target vehicle;

[0249] An input module 1104 is configured to input the target image into a camera positioning model to obtain a device pose of the target camera, wherein the camera positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the labeled device pose associated with the sample images based on a pose feature extraction model;

[0250] The control module 1106 is configured to determine the driving trajectory of the target vehicle according to the device posture of the target shooting device, and control the target vehicle according to the driving trajectory.

[0251] To sum up, in the above-mentioned device, when training the shooting device positioning model, a rendered image associated with the sample image is introduced. The shooting device positioning model is trained based on the rendered image, and the rendered image associated with the sample image is determined by the pose feature extraction model. The efficient reasoning based on the pose feature extraction model can improve the model training speed and reasoning accuracy. Furthermore, in the process of model application, the device pose obtained by the shooting device positioning model according to the given target image is more accurate, ensuring that the pose estimation process is simple, reducing the reasoning cost, and at the same time ensuring the accuracy and precision of the pose estimation results.

[0252] The above is a schematic diagram of a vehicle control device according to this embodiment. It should be noted that the technical solution of this vehicle control device and the technical solution of the aforementioned method for determining the position and posture of a camera are based on the same concept. For details not described in detail in the technical solution of the vehicle control device, please refer to the description of the technical solution of the aforementioned method for determining the position and posture of a camera.

[0253] Figure 12 The following is a block diagram of a computing device 1200 according to one embodiment of the present disclosure. Components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0254] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0255] In one embodiment of the present application, the above components of the computing device 1200 and Figure 12 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 12 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0256] Computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1200 may also be a mobile or stationary server.

[0257] The processor 1220 is configured to execute the following computer-executable instructions, which implement the steps of the above method when executed by the processor.

[0258] The above is a schematic solution of a computing device of this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above method.

[0259] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above method when executed by a processor.

[0260] The above is a schematic solution of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above method belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above method.

[0261] An embodiment of the present specification further provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0262] The above is an illustrative solution of a computer program product of this embodiment. It should be noted that the technical solution of the computer program product and the technical solution of the above method are of the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the above method.

[0263] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0264] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0265] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0266] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0267] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a camera positioning model, comprising: Determine first sample images of different perspectives and tag device poses corresponding to the first sample images, and determine a first rendered image associated with the first sample image, wherein the first rendered image is obtained by processing the tag device pose associated with the sample image according to a pose feature extraction model; Inputting the first sample image into a shooting device positioning model to obtain first sample image features and a first predicted device pose associated with the first sample image; Inputting the first rendered image into the shooting device positioning model to obtain first rendered image features; The shooting device positioning model is trained according to the first sample image feature, the first rendered image feature, the first predicted device pose and the label device pose until a shooting device positioning model that meets a first training stop condition is obtained.

2. The method for training a camera positioning model according to claim 1, wherein determining the first rendered image associated with the first sample image comprises: Inputting the tag device posture associated with the first sample image into the posture feature extraction model; In the posture feature extraction model, the posture of the tag device is processed to obtain image parameters of the first sample image; Perform volume rendering processing on the image parameters to obtain a first rendered image associated with the first sample image.

3. The method for training a camera positioning model according to claim 2, wherein the image parameters include an opacity parameter and a color parameter; The performing volume rendering processing on the image parameters to obtain a first rendered image associated with the first sample image includes: Volume rendering is performed on the opacity parameter and the color parameter to obtain a first rendered image associated with the first sample image.

4. The method for training a camera positioning model according to claim 3, further comprising: after inputting the label device pose associated with the first sample image into the pose feature extraction model; In the posture feature extraction model, processing the posture of the tag device to obtain a posture embedding vector associated with the first sample image; performing volume rendering on the opacity parameter and the pose embedding vector to obtain a first predicted device pose feature associated with the first sample image; Determine a second predicted device pose associated with the first sample image according to the first predicted device pose feature.

5. The method for training a camera positioning model according to claim 4, further comprising: after determining the second predicted device pose associated with the first sample image; The posture feature extraction model is trained according to the label device posture and the second predicted device posture.

6. The method for training a camera positioning model according to claim 4, further comprising: performing volume rendering on the opacity parameter and the pose embedding vector to obtain a first predicted device pose feature associated with the first sample image; Inputting the first predicted device posture into the posture feature extraction model to obtain a second predicted device posture feature associated with the first sample image; The posture feature extraction model is trained according to the first predicted device posture feature and the second predicted device posture feature.

7. The method for training a camera positioning model according to claim 1, wherein the training of the camera positioning model according to the first sample image feature, the first rendered image feature, the first predicted device pose, and the labeled device pose until a camera positioning model that satisfies a first training stop condition is obtained comprises: Calculating a first model loss value according to the first sample image feature and the first rendered image feature; Calculating a second model loss value according to the first predicted device pose and the tag device pose; The shooting device positioning model is trained according to the first model loss value and the second model loss value until a shooting device positioning model that meets a first training stop condition is obtained.

8. The method for training a camera positioning model according to claim 1, wherein the method further comprises: training the camera positioning model according to the first sample image feature, the first rendered image feature, the first predicted device pose, and the labeled device pose until a camera positioning model that satisfies a first training stop condition is obtained; determining a second sample image, and determining a second rendered image associated with the second sample image; Inputting the second sample image into the shooting device positioning model to obtain second sample image features; Inputting the second rendered image into the shooting device positioning model to obtain second rendered image features; The shooting device positioning model is trained according to the second sample image feature and the second rendered image feature until a shooting device positioning model that meets a second training stop condition is obtained.

9. The method for training a camera positioning model according to claim 8, wherein determining the second rendered image associated with the second sample image comprises: Inputting the second sample image into the shooting device positioning model to obtain a third predicted device pose associated with the second sample image; The third predicted device posture is input into the posture feature extraction model to obtain a second rendered image associated with the second sample image.

10. The method for training a camera positioning model according to claim 9, further comprising: after inputting the second rendered image into the camera positioning model; obtaining a fourth predicted device pose associated with the second rendered image; Accordingly, the method further comprises: Inputting the third predicted device posture into the posture feature extraction model to obtain a third predicted device posture feature; Inputting the fourth predicted device posture into the posture feature extraction model to obtain a fourth predicted device posture feature; The posture feature extraction model is trained according to the third predicted device posture feature and the fourth predicted device posture feature.

11. A method for determining a position and posture of a camera device, comprising: Determine a target image captured by a target shooting device; The target image is input into a shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on a posture feature extraction model.

12. The method for determining the position and posture of a photographing device according to claim 11, wherein determining the target image captured by the target photographing device comprises: receiving a target image sent by a target shooting device, wherein the target shooting device is disposed on a target vehicle; Accordingly, after obtaining the device posture of the target shooting device, the method further includes: The driving trajectory of the target vehicle is determined according to the device posture of the target shooting device, and the target vehicle is controlled according to the driving trajectory.

13. The method for determining the position and posture of a photographing device according to claim 11, wherein the step of training the photographing device positioning model comprises: Determine first sample images of different perspectives and tag device poses corresponding to the first sample images, and determine a first rendered image associated with the first sample images; Inputting the first sample image into a shooting device positioning model to obtain first sample image features and a first predicted device pose associated with the first sample image; Inputting the first rendered image into the shooting device positioning model to obtain first rendered image features; The shooting device positioning model is trained according to the first sample image feature, the first rendered image feature, the first predicted device pose and the label device pose until a shooting device positioning model that meets a first training stop condition is obtained.

14. A method for determining the position and posture of a camera device, applied to a cloud-side device, comprising: Receiving a target image sent by a target shooting device; The target image is input into a shooting device positioning model to obtain the device posture of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the label device posture associated with the sample images based on a posture feature extraction model.

15. A vehicle control method, applied to a cloud-side device, comprising: receiving a target image sent by a target shooting device, wherein the target shooting device is disposed on a target vehicle; Inputting the target image into a shooting device positioning model to obtain a device pose of the target shooting device, wherein the shooting device positioning model is trained based on sample images of different perspectives and rendered images associated with the sample images, and the rendered images are obtained by processing the labeled device pose associated with the sample images based on a pose feature extraction model; The driving trajectory of the target vehicle is determined according to the device posture of the target shooting device, and the target vehicle is controlled according to the driving trajectory.

16. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 15 are implemented.

17. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

18. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 15.

Images