An image processing method and related device

Abstract

The embodiment of the application discloses an image processing method and related equipment, which can be used in a three-dimensional scene in the field of artificial intelligence. The method comprises the following steps: inputting a training sample into a first model to obtain feature information of each image in the training sample, and training the first model; the training sample comprises images of a first scene under multiple viewing angles, wherein the images comprise a first image and a second image; a training target comprises improving the similarity between first feature information and second feature information; the first feature information comprises feature information of a first point in the first image; the second feature information comprises feature information of a second point in the second image; the first point and the second point correspond to the same point in the first scene. When the trained first model is used for feature extraction, the feature information of the same point in the scene under different viewing angles is more similar, the feature information can reflect the relationship between images under different viewing angles, and the geometric perception ability of the model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image processing method and related equipment. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] In recent years, using artificial intelligence technology to perform various image processing tasks in three-dimensional (3D) scenes has become a common application of artificial intelligence. Examples include 3D scene reconstruction, 3D image classification, 3D image object detection, 3D image semantic segmentation, and other image processing tasks.

[0004] Specifically, multiple images of the same scene from multiple perspectives can be obtained, with each image representing the scene from one perspective. These multiple images are then input into a model, which extracts features from each image from each perspective to obtain the feature information of each image. Finally, a prediction result corresponding to the image processing task is generated based on the feature information of each image.

[0005] However, since the model extracts features from images from each viewpoint separately, it does not learn the relationships between images from different viewpoints. Therefore, a feature extraction scheme that can obtain more information is urgently needed. Summary of the Invention

[0006] This application provides an image processing method and related equipment. The training objective includes improving the similarity between feature information of the same point in images from different viewpoints in a first scene. Through the aforementioned scheme, when using the trained first model to extract features from images of the same scene from different viewpoints, the feature information of the same point in the scene from different viewpoints will be more similar. Thus, the feature information obtained by the aforementioned first model can reflect the connection between images from different viewpoints, which is beneficial to obtaining richer information in the feature extraction stage. In addition, the model's ability to integrate information from multiple viewpoints is also beneficial to enhancing the model's geometric perception capability.

[0007] To address the aforementioned technical problems, this application provides the following technical solutions:

[0008] In a first aspect, embodiments of this application provide a training method for a model that can be used in a three-dimensional scene in the field of artificial intelligence. The method includes: a training device acquiring training samples, wherein multiple images in the training samples include images of a first scene from multiple perspectives; exemplarily, the first scene can be an environment, a person, an animal, an object, or other types of first scenes, for example, "environment" can be a traffic road environment, an indoor environment, a park environment, or other types of environments; "person" can be a virtual character or a real person; "animal" can be a virtual animal or a real animal, etc., without limitation here.

[0009] The training device inputs training samples into a first model, extracts features through the first model to obtain feature information for each image in the training samples, and trains the first model based on the feature information of each image in the training samples and a first loss term. The multiple images include a first image and a second image. The goal of training with the first loss term is to improve the similarity between the first feature information and the second feature information. The first feature information includes the feature information of a first point in the first image, and the second feature information includes the feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. In other words, projecting the same point in the three-dimensional first scene onto images from different viewpoints yields the first point in the first image and the second point in the second image, respectively. Therefore, the purpose of training with the first loss term is to improve the similarity between the feature information of points with the same semantics in images from different viewpoints.

[0010] In this implementation, a first loss term is used during training. The goal of training the first model using the first loss term includes improving the similarity between the first feature information and the second feature information. The first feature information includes the feature information of a first point in the first image, and the second feature information includes the feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. That is, the training goal includes improving the similarity between the feature information of the same point in the first scene in images from different perspectives. Through the aforementioned scheme, when using the trained first model to extract features from images of the same scene from different perspectives, the feature information of the same point in the scene in images from different perspectives will be more similar. Therefore, the feature information obtained by the aforementioned first model can reflect the connection between images from different perspectives, which is beneficial to obtaining richer information in the feature extraction stage. In addition, the model's ability to integrate information from multiple perspectives is also beneficial to enhancing the model's geometric perception capability.

[0011] In one possible implementation, the training device trains a first model based on the feature information of each image in the training samples and a first loss term, including: the training device trains the first model based on the feature information of each image in the training samples, the first loss term, and a second loss term, wherein the objective of training using the second loss term includes reducing the similarity between the first feature information and the third feature information, the third feature information including the feature information of a third point in the second image, the third point being different from the second point.

[0012] In this implementation, since the first point in the first image and the second point in the second image correspond to the same point in the first scene, and the third point is different from the second point, that is, the third point and the first point correspond to different points in the first scene, the purpose of training with the second loss term is to reduce the similarity between the first feature information of the first point and the third feature information of the third point. That is, not only will the feature information of the same point in the first scene be more similar in images from different perspectives, but the feature information of different points in the first scene will be less similar in images from different perspectives. Therefore, the feature information obtained by the aforementioned first model can reflect more connections between images from different perspectives, which is conducive to further enhancing the geometric perception ability of the model, thereby improving the accuracy of the prediction results output by the first model.

[0013] In one possible implementation, both the second and third points are located on the first line. Projecting the second line onto the second image yields the first line. The second line passes through the first point and also passes through the focal point of the camera capturing the first image and / or the origin of the camera coordinate system corresponding to the first image. In this embodiment, after determining the first point, the second line can be obtained. Projecting the second line onto images from other perspectives (i.e., the second image) allows the determination of the first line in the second image. Points with the same semantic meaning as the first point are generally located on the first line. Sampling the third point from the first line helps the first model learn which points in images from other perspectives are consistent with the points in the first scene corresponding to the first point, and which points in images from other perspectives are inconsistent with the points in the first scene corresponding to the first point. This helps to further improve the accuracy of the prediction results output by the first model.

[0014] In one possible implementation, the feature information of each image in the training samples includes updated feature information of multiple first pixels in each image of the training samples. The training device performs feature extraction using a first model to obtain the feature information of each image in the training samples, including: the training device acquiring initial feature information of second pixels, where the first and second pixels are points in different images among multiple images, and the first and second pixels have the same semantics; and the training device fusing the initial feature information of the first pixels with the initial feature information of the second pixels to obtain updated feature information of the first pixels. Optionally, the feature information of each image in the training samples may also include initial feature information of a fourth pixel in each image, where the fourth pixel and the first pixel are different pixels in the same image.

[0015] In this implementation, during the feature extraction stage of the first model, after obtaining the initial feature information of the first pixel in the image of the first scene from a certain perspective, the feature information of the second pixel can be obtained from the images of the first scene from other perspectives. The first pixel and the second pixel have the same semantics. The initial feature information of the second pixel is fused with the initial feature information of the first pixel to obtain the updated feature information of the first pixel, which is also the updated feature information of the first pixel included in the feature information of each image in the training samples. Through the aforementioned method, the updated feature information of the first pixel includes the feature information of a point in space from multiple perspectives. Thus, the feature information of each image can reflect the geometric constraints between images from different perspectives, which is beneficial to obtaining richer information and improving the accuracy of the prediction results output by the first model.

[0016] In one possible implementation, the first pixel is included in the third image of the training samples. The training device acquires the initial feature information of the second pixel by: acquiring at least one third point on a third line, the third line passing through the first pixel, and the third line also passing through the focal point of the camera capturing the third image and / or the origin of the image coordinate system corresponding to the third image; the training device acquires a feature information set, which includes the initial feature information of each of multiple projection points, where each projection point is a point that projects the third point onto a fourth image, and the fourth image is an image other than the third image in the training samples. The multiple projection points include the second pixel, meaning the feature information set includes the initial feature information of the second pixel. The training device then fuses the initial feature information of the first pixel with the initial feature information of the second pixel, including: fusing the initial feature information of the first pixel with the feature information set. This implementation provides a simple scheme for acquiring the initial feature information of the second pixel, reducing the implementation difficulty while ensuring accurate acquisition of the initial feature information of the second pixel.

[0017] In one possible implementation, the training samples also include a new perspective. After the training device extracts features using a first model to obtain feature information for each image in the training samples, the method further includes: the training device performs feature processing using the first model based on the feature information of each image in the training samples to obtain a predicted image of the first scene from the new perspective. The training device trains the first model based on the feature information of each image in the training samples and a first loss term, including: the training device trains the first model based on the feature information of each image in the training samples, the predicted image, the first loss term, and a third loss term, where the third loss term indicates the similarity between the predicted image and the expected image of the first scene from the new perspective. This implementation provides a specific application scenario for this solution, improving its integration with real-world applications and reducing its implementation difficulty.

[0018] In one possible implementation, the predicted image includes first color values ​​of multiple pixels. The training device performs feature processing using a first model to obtain a predicted image of the first scene from a new perspective. This includes: the training device performing feature processing using the first model to obtain information generated by the first model corresponding to a third pixel, where the third pixel is any one of the multiple pixels. The information corresponding to the third pixel includes multiple second color values ​​and a voxel density corresponding to each second color value. For example, the information corresponding to the third pixel includes the second color values ​​and voxel densities of multiple sampling points, where all sampling points corresponding to the third pixel are located on a fourth line. The fourth line originates from the origin of the camera coordinate system corresponding to the predicted image and passes through the third pixel. The training device normalizes the voxel density corresponding to each second color value to obtain a weight for each second color value, and performs a weighted sum of the multiple second color values ​​based on their weights to obtain the first color value of the third pixel. For example, the training device normalizes the voxel density corresponding to each sampling point to obtain the weight of the second color value for that sampling point.

[0019] In this implementation, the voxel density corresponding to each second color value is normalized to obtain the weight of each second color value. Then, based on the weight of each second color value, multiple second color values ​​are weighted and summed to obtain the first color value of the third pixel. The above scheme provides a method for obtaining the color value of pixels in the predicted image, and the above scheme is simple and convenient.

[0020] Secondly, embodiments of this application provide a model training method that can be used in three-dimensional scenes in the field of artificial intelligence. The method includes: an execution device acquiring data to be processed, wherein multiple images in the data to be processed include images of the scene from multiple perspectives; inputting the data to be processed into a first model, performing feature extraction through the first model to obtain feature information of each image in the data to be processed; and performing feature processing through the first model based on the feature information of each image in the data to be processed to obtain a prediction result output by the first model.

[0021] The first model is trained based on training samples and a first loss term. The training samples include multiple images of the first scene from multiple perspectives. The training samples include a first image and a second image. The first loss term indicates the similarity between first feature information and second feature information. The first feature information includes feature information of a first point in the first image. The second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0022] In one possible implementation, the first model is trained based on training samples, a first loss term, and a second loss term. The goal of training with the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of the first pixel in the second image, and the first pixel and the second pixel are different.

[0023] In one possible implementation, the feature information of each image in the data to be processed includes the feature information of multiple first pixels of each image in the data to be processed. The execution device performs feature extraction through a first model to obtain the feature information of each image in the data to be processed, including: the execution device acquiring the initial feature information of the second pixel, where the first pixel and the second pixel are points in different images in the data packet to be processed, and the first pixel and the second pixel have the same semantics; and fusing the initial feature information of the first pixel and the initial feature information of the second pixel to obtain the feature information of the first pixel.

[0024] In the second aspect of this application, the execution device can also be used to execute the steps of the training device in the first aspect and various possible implementations of the first aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the various possible implementations of the second aspect can all be found in the first aspect, and will not be repeated here.

[0025] Thirdly, embodiments of this application provide a model training method that can be used in three-dimensional scenes in the field of artificial intelligence. The method includes: an execution device acquiring data to be processed, wherein multiple images in the data to be processed include images of the scene from multiple perspectives; inputting the data to be processed into a first model, performing feature extraction through the first model to obtain feature information of each image in the data to be processed; and performing feature processing through the first model based on the feature information of each image in the data to be processed to obtain a prediction result output by the first model.

[0026] The feature information of each image in the data to be processed includes the feature information of multiple first pixels of each image in the data to be processed. Feature extraction is performed by the first model to obtain the feature information of each image in the data to be processed, including: obtaining the initial feature information of the second pixel, where the first pixel and the second pixel are points in different images in the data to be processed, and the first pixel and the second pixel have the same semantics; and fusing the initial feature information of the first pixel and the initial feature information of the second pixel to obtain the feature information of the first pixel.

[0027] In one possible implementation, the first model is trained based on training samples and a first loss term. The training samples include multiple images of the first scene from multiple perspectives. The training samples include a first image and a second image. The first loss term indicates the similarity between first feature information and second feature information. The first feature information includes feature information of a first point in the first image. The second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0028] In one possible implementation, the first model is trained based on training samples, a first loss term, and a second loss term. The goal of training with the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of the first pixel in the second image, and the first pixel and the second pixel are different.

[0029] In one possible implementation, both the second and third points are located on the first line. The second line is projected onto the second image to obtain the first line. The second line passes through the first point and also passes through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image.

[0030] In one possible implementation, the first pixel is included in the third image of the data to be processed. The execution device acquires initial feature information of the second pixel, including: acquiring at least one third point on a third line, the third line passing through the first pixel, and the third line also passing through the focal point of the camera capturing the third image and / or the origin of the image coordinate system corresponding to the third image; the execution device acquires a feature information set, the feature information set including initial feature information of each of a plurality of projection points, the projection point being the point that projects the third point onto a fourth image, the fourth image being an image other than the third image in the data to be processed, and the plurality of projection points including the second pixel. The execution device fuses the initial feature information of the first pixel with the initial feature information of the second pixel, including: fusing the initial feature information of the first pixel with the feature information set.

[0031] In one possible implementation, the data to be processed also includes a new perspective. After the execution device extracts features through the first model to obtain feature information of each image in the data to be processed, the method further includes: the execution device performs feature processing through the first model based on the feature information of each image in the data to be processed to obtain a predicted image of the first scene under the new perspective.

[0032] In one possible implementation, the predicted image includes first color values ​​of multiple pixels. The execution device performs feature processing through a first model to obtain a predicted image of the first scene from a new perspective. This includes: the execution device performing feature processing through the first model to obtain information generated by the first model corresponding to a third pixel, where the third pixel is any one of the multiple pixels, and the information corresponding to the third pixel includes multiple second color values ​​and a voxel density corresponding to each second color value; the execution device normalizes the voxel density corresponding to each second color value to obtain a weight for each second color value; and the execution device performs a weighted summation of the multiple second color values ​​according to the weight of each second color value to obtain the first color value of the third pixel.

[0033] For details on the specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in each possible implementation of the third aspect, please refer to the first aspect; they will not be repeated here.

[0034] Fourthly, embodiments of this application provide a model training apparatus that can be used in a three-dimensional scene in the field of artificial intelligence. The model training apparatus includes: an acquisition module for acquiring training samples, wherein multiple images in the training samples include images of a first scene from multiple perspectives; a processing module for inputting the training samples into a first model and performing feature extraction through the first model to obtain feature information of each image in the training samples; and a training module for training the first model based on the feature information of each image in the training samples and a first loss term.

[0035] The multiple images include a first image and a second image. The goal of training using a first loss term is to improve the similarity between the first feature information and the second feature information. The first feature information includes the feature information of a first point in the first image, and the second feature information includes the feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0036] In the fourth aspect of this application, the training device for the model can also be used to execute the steps performed by the training device in the first aspect and various possible implementations of the first aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the various possible implementations of the fourth aspect can all be found in the first aspect, and will not be repeated here.

[0037] Fifthly, embodiments of this application provide an image processing apparatus that can be used in three-dimensional scenes in the field of artificial intelligence. The image processing apparatus includes: an acquisition module for acquiring data to be processed, wherein multiple images in the data to be processed include images of the scene from multiple perspectives; a processing module for inputting the data to be processed into a first model and performing feature extraction through the first model to obtain feature information of each image in the data to be processed; and the processing module is further configured to perform feature processing through the first model based on the feature information of each image in the data to be processed to obtain a prediction result output by the first model.

[0038] The first model is trained based on training samples and a first loss term. The training samples include multiple images of the first scene from multiple perspectives. The training samples include a first image and a second image. The first loss term indicates the similarity between first feature information and second feature information. The first feature information includes feature information of a first point in the first image. The second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0039] In the fifth aspect of this application, the image processing apparatus can also be used to perform the steps executed by the execution device in the second aspect and various possible implementations of the second aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the various possible implementations of the fifth aspect can be found in the second aspect, and will not be repeated here.

[0040] Sixthly, embodiments of this application provide an image processing apparatus that can be used in three-dimensional scenes in the field of artificial intelligence. The image processing apparatus includes: an acquisition module for acquiring data to be processed, wherein multiple images in the data to be processed include images of the scene from multiple perspectives; a processing module for inputting the data to be processed into a first model and performing feature extraction through the first model to obtain feature information of each image in the data to be processed; and the processing module is further configured to perform feature processing through the first model based on the feature information of each image in the data to be processed to obtain a prediction result output by the first model.

[0041] The feature information of each image in the data to be processed includes the feature information of multiple first pixels in each image in the data to be processed; the processing module is specifically used to obtain the initial feature information of the second pixel, and fuse the initial feature information of the first pixel with the initial feature information of the second pixel to obtain the feature information of the first pixel, wherein the first pixel and the second pixel are points in different images in the data to be processed, and the first pixel and the second pixel have the same semantics.

[0042] In the sixth aspect of this application, the image processing apparatus can also be used to perform the steps executed by the execution device in the third aspect and various possible implementations of the third aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the various possible implementations of the sixth aspect can be found in the third aspect, and will not be repeated here.

[0043] In a seventh aspect, embodiments of this application provide a computer program product, which includes a program that, when run on a computer, causes the computer to perform the methods described in the first, second, or third aspects above.

[0044] Eighthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first, second, or third aspects above.

[0045] Ninthly, embodiments of this application provide an execution device, including a processor and a memory, the processor being coupled to the memory, the memory being used to store a program; the processor being used to execute the program in the memory, causing the execution device to perform the image processing method described in the second or third aspect above.

[0046] In a tenth aspect, embodiments of this application provide a training device, including a processor and a memory, wherein the processor is coupled to the memory, the memory is used to store a program, and the processor is used to execute the program in the memory, such that the training device performs the training method of the model described in the first aspect.

[0047] Eleventhly, this application provides a chip system including a processor for supporting a terminal device or communication device in implementing the functions involved in the above aspects, such as transmitting or processing data and / or information involved in the above methods. In one possible design, the chip system further includes a memory for storing necessary program instructions and data for the terminal device or communication device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0048] Figure 1a A schematic diagram of the main framework of artificial intelligence provided in the embodiments of this application;

[0049] Figure 1b A schematic diagram of a scene requiring three-dimensional reconstruction, provided for an embodiment of this application;

[0050] Figure 2 A system architecture diagram of an image processing system provided in an embodiment of this application;

[0051] Figure 3 A schematic flowchart illustrating a training method for a model provided in an embodiment of this application;

[0052] Figure 4 A schematic diagram of the first and second points provided in the embodiments of this application;

[0053] Figure 5 Another flowchart illustrating the training method for the model provided in the embodiments of this application;

[0054] Figure 6 A schematic diagram of multiple projection points provided in this application;

[0055] Figure 7 Another schematic diagram of the multiple projection points provided in this application;

[0056] Figure 8 A schematic diagram of the fourth line provided in an embodiment of this application;

[0057] Figure 9 A schematic diagram illustrating the first, second, and third points provided in an embodiment of this application;

[0058] Figure 10 A schematic diagram illustrating the calculation process of the loss function provided in the embodiments of this application;

[0059] Figure 11 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0060] Figure 12A schematic diagram of a training device for a model provided in an embodiment of this application;

[0061] Figure 13 A schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0062] Figure 14 A schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0063] Figure 15 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0064] Figure 16 A schematic diagram of the structure of a training device provided in an embodiment of this application;

[0065] Figure 17 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0066] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0067] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0068] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1a , Figure 1aThe diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0069] (1) Infrastructure

[0070] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically employ hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0071] (2) Data

[0072] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0073] (3) Data processing

[0074] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0075] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0076] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0077] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0078] (4) General ability

[0079] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0080] (5) Smart Products and Industry Applications

[0081] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, autonomous driving, and smart cities.

[0082] This application can be applied to various application fields of artificial intelligence technology, specifically for processing images in 3D scenes in various application fields. The following are examples of several application scenarios of this application.

[0083] Application Scenario 1: Generating images from new perspectives

[0084] The electronic device can acquire a set of images corresponding to the same scene, including multiple images of the aforementioned scene from multiple known viewpoints; the electronic device synthesizes an image from a new viewpoint based on the images of the aforementioned scene from multiple known viewpoints, so as to achieve three-dimensional reconstruction of the scene.

[0085] For example, in the field of smart terminals, there is a need to reconstruct 3D virtual digital humans (i.e., an example of the "scenarios mentioned above"). In order to improve the quality of the reconstructed 3D virtual digital humans, images of the 3D virtual digital humans from multiple new perspectives can be generated based on the images of the 3D virtual digital humans from multiple known perspectives, and then the 3D virtual digital humans can be reconstructed in three dimensions.

[0086] For example, in the field of autonomous driving, there is a need to reconstruct the three-dimensional environment around the vehicle. Based on images of the vehicle's surroundings from multiple known perspectives, images of the vehicle's surroundings from multiple new perspectives can be generated, thereby enabling the three-dimensional reconstruction of the vehicle's surroundings.

[0087] To understand this solution more intuitively, please refer to [link / reference]. Figure 1b , Figure 1b This is a schematic diagram of a scene requiring three-dimensional reconstruction, provided in an embodiment of this application. Figure 1b The image shown depicts the vehicle's surrounding environment, which requires 3D reconstruction. It should be understood that... Figure 1b The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0088] Application Scenario 2: Image Classification

[0089] For example, after an electronic device acquires a 3D image of a scene, it can acquire images of the scene from multiple perspectives from the 3D image of the scene, and classify the objects in the scene based on the images of the scene from multiple perspectives.

[0090] Application Scenario 3: Object Detection

[0091] For example, in the field of autonomous driving, autonomous vehicles can use sensors to collect images of the environment around the vehicle (i.e., an example of the "same scene") from multiple perspectives, and perform target detection based on the collected images from the aforementioned multiple perspectives to obtain prediction results corresponding to the images from the aforementioned multiple perspectives. These prediction results are used to indicate the position of at least one object in the environment around the vehicle, and the autonomous vehicle can plan its driving path based on the aforementioned prediction results.

[0092] It should be noted that the aforementioned vehicles may be cars, trucks, motorcycles, buses, ships, airplanes, helicopters, lawnmowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, or trains, etc., and this application embodiment does not impose any special limitations.

[0093] For example, in the field of intelligent surveillance, many cameras are installed in public places and on traffic roads. After a few intelligent cameras collect images of the surrounding environment from multiple perspectives, they can perform target detection tasks based on the images from the aforementioned multiple perspectives.

[0094] For example, in the field of smart homes, mobile robots (such as robot vacuum cleaners, tutoring robots, or other mobile robots) can collect images of the robot's surrounding environment from multiple perspectives, and perform target detection based on the collected images from the aforementioned multiple perspectives to obtain prediction results corresponding to the images from the aforementioned multiple perspectives. These prediction results are used to indicate the location of at least one obstacle around the mobile robot.

[0095] Application Scenario 4: Semantic segmentation of 3D images

[0096] For example, after acquiring a 3D image of a scene, an electronic device can obtain images of the scene from multiple viewpoints and generate prediction results based on the images of the scene from multiple viewpoints. The prediction results are used to indicate the category of all pixels in the image of the scene from each viewpoint.

[0097] It should be noted that the examples of various application scenarios mentioned above are only for the convenience of understanding this solution. This application can also be used in other application scenarios, which are not exhaustively listed here. In all the above application scenarios, after acquiring images of the same scene from multiple perspectives, these images can be input into a machine learning model. The solution provided in this application is used to extract features from the images from the aforementioned multiple perspectives to obtain feature information for each image. Then, prediction results can be generated based on the feature information of each image, which is beneficial for obtaining richer information in the feature extraction stage and also helps to improve the accuracy of the final output prediction results.

[0098] Before providing a detailed description of the methods provided in the embodiments of this application, please refer to [the relevant documentation / reference]. Figure 2 , Figure 2 This is a system architecture diagram of an image processing system provided in an embodiment of this application. Figure 2 In the image processing system 200, there are training devices 210, database 220, execution devices 230, data storage systems 240 and client devices 250, and the execution devices 230 include computing modules 231.

[0099] The database 220 stores a training dataset, which includes multiple training samples. Each training sample contains multiple images of a first scene from multiple viewpoints. During the training phase of the first model 201, the training device 210 generates the first model 201 and iteratively trains it using the training dataset to obtain the trained first model 201. The first model 201 can be specifically represented as a neural network or as a non-neural network machine learning model; for example, the first model 201 can be a convolutional neural network, a residual neural network, a transform model, or other types of models, etc., which are not exhaustively listed here.

[0100] During the training process of the training device 210 on the first model 201, please refer to [the relevant documentation]. Figure 3 , Figure 3 This is a flowchart illustrating a training method for a model provided in an embodiment of this application. 301. The training device 210 acquires training samples, including multiple images of a first scene from multiple viewpoints.

[0101] 302. The training device 210 inputs the training samples into the first model, performs feature extraction through the first model, and obtains the feature information of each image in the training samples.

[0102] 303. The training device 210 trains the first model based on the feature information of each image in the training samples and the first loss term; wherein, the multiple images include the first image and the second image, and the goal of training with the first loss term is to improve the similarity between the first feature information and the second feature information. The first feature information includes the feature information of the first point in the first image, and the second feature information includes the feature information of the second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene (hereinafter referred to as the "target point" for ease of description).

[0103] For example, "the first point in the first image and the second point in the second image correspond to the same target point in the first scene" can be understood as the first point being the point obtained when observing the target point from the first perspective, and the second point being the point obtained when observing the target point from the second perspective. That is, the first point in the first image and the second point in the second image have the same semantics.

[0104] For a more intuitive understanding of this solution, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of the first and second points provided in an embodiment of this application. Figure 4 Includes two sub-diagrams, left and right. Figure 4 The left and right sub-plots respectively illustrate the first scene (i.e. Figure 4The images shown are of a three-dimensional indoor environment viewed from two different perspectives. Figure 4 The target point is a point on the back of a chair in the first scene. Figure 4 The left sub-illustration shows the first point in the first image corresponding to the target point. Figure 4 The right sub-illustration shows the second point in the second image corresponding to the target point. It should be understood that... Figure 4 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0105] It should be noted that, due to possible deviations in obtaining the first and second feature information, the points in the three-dimensional space corresponding to the first point and the points in the three-dimensional space corresponding to the second point may or may not completely overlap, for example, there may be a short distance between the two points mentioned above.

[0106] In this embodiment of the application, the training objective includes improving the similarity between feature information of the same point in images from different perspectives in the first scene. Through the aforementioned scheme, when the trained first model is used to extract features from images of the same scene from different perspectives, the feature information of the same point in the scene from different perspectives will be more similar. Thus, the feature information obtained by the aforementioned first model can reflect the connection between images from different perspectives, which is beneficial to obtaining richer information in the feature extraction stage. In addition, the model's ability to integrate information from multiple perspectives is also beneficial to enhancing the model's geometric perception capability.

[0107] The first trained model 201 obtained by the training device 210 can be applied to the computing module 231 of the execution device 230. For example, the execution device 230 can be a mobile phone, tablet, laptop, virtual reality (VR) device, monitoring system, radar data processing system or other types of devices, etc.

[0108] During the inference phase of the first model 201, the execution device 230 can input the data to be processed into the first model 201 of the calculation module 231, so that the first model 201 can process the data to be processed and obtain the prediction result output by the first model 201. The execution device 230 can call data, code, etc., from the data storage system 240, and can also store data, instructions, etc., in the data storage system 240. The data storage system 240 can be located within the execution device 230, or it can be an external memory relative to the execution device 230.

[0109] In some embodiments of this application, please refer to Figure 2The execution device 230 and the client device 250 can be separate independent devices. The execution device 230 is configured with an input / output (I / O) interface to interact with the client device 250. The "user" can input data to be processed through the client device 250. The client device 250 sends the data to be processed to the execution device 230 through the I / O interface. After the execution device 230 generates the prediction decision information corresponding to the data to be processed through the first machine learning model / rule 201 in the calculation module 231, it can return the aforementioned prediction decision information to the client device 250 through the I / O interface and provide it to the user.

[0110] It is worth noting that Figure 2 This is merely a schematic diagram of an image processing system provided in an embodiment of the present invention, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in some other embodiments of this application, the execution device 230 may be configured in the client device 250. As an example, when the client device is a mobile phone or tablet, the execution device 230 may be a module in the host CPU of the mobile phone or tablet used for array image processing. The execution device 230 may also be a graphics processing unit (GPU) or neural network processor (NPU) in the mobile phone or tablet, with the GPU or NPU mounted on the host processor as a coprocessor, and the host processor assigning tasks.

[0111] Based on the above description, the specific implementation process of the training and application phases of the image processing method provided in the embodiments of this application will now be described.

[0112] I. Training Phase

[0113] In this embodiment, the training phase describes how the training device 210 uses the training data set maintained in the database 220 to train the first model 201. For details, please refer to... Figure 5 , Figure 5 This is another schematic flowchart illustrating the training method for the model provided in this application embodiment. The training method for the model provided in this application embodiment may include:

[0114] 501. Obtain training samples. The training samples include multiple images of the first scene from multiple perspectives.

[0115] In this embodiment, the training device can acquire a training data set, which includes multiple training samples and the expected result corresponding to each training sample. Each training sample includes multiple images, which include images of the first scene from multiple perspectives. For example, the first scene can be an environment, a person, an animal, an object, or other types of first scenes. For instance, "environment" can be a traffic road environment, an indoor environment, a park environment, or other types of environments; "person" can be a virtual character or a real person; "animal" can be a virtual animal or a real animal. The specific representation of the first scene can be flexibly determined based on the actual situation, and is not limited in this embodiment.

[0116] For example, the task performed by the first model can be any of the following: generating images from a new perspective, image classification, object detection, semantic segmentation of images, or other tasks in a three-dimensional scene, etc., which will not be listed here.

[0117] The specific form of the expected result corresponding to each training sample needs to be determined in conjunction with the type of task performed by the first model. For example, if the task of the first model is to generate an image of the first scene from a new perspective based on images of the first scene from multiple known perspectives, then the "training sample" may also include the new perspective, and the "expected result corresponding to the training sample" may be the correct image of the first scene from the aforementioned new perspective, or it may be called the expected image of the first scene from the aforementioned new perspective.

[0118] For example, if the task of the first model is to classify the first scene based on images of the first scene from multiple perspectives, then the "expected result corresponding to the training samples" can be represented as the correct category of the first scene. For another example, if the task of the first model is to detect objects in the first scene based on images of the first scene from multiple perspectives, then the "expected result corresponding to the training samples" can be represented as the correct location information of at least one object in the first scene.

[0119] For example, if the task of the first machine learning model is to perform semantic segmentation on images of the first scene from multiple perspectives, then the "expected result corresponding to the training samples" can be represented as the correct category of each pixel in each of the aforementioned multiple images, etc. It should be understood that the examples here are only for the convenience of understanding this scheme and are not intended to limit this scheme.

[0120] 502. Input the training samples into the first model, and perform feature extraction through the first model to obtain the feature information of each image in the training samples.

[0121] In this embodiment of the application, the training device can perform step 502 in a variety of ways. In one implementation, step 502 may include: the training device can extract features from each image in the training samples using the first model to obtain the initial feature information of each image, that is, to obtain the initial feature information of each pixel in each image.

[0122] The training device can update the initial feature information of multiple first pixels in each image through the first model based on the initial feature information of each image in the training samples, and obtain the updated feature information of each first pixel. The feature information of each image in the training samples can include the updated feature information of each first pixel in each image, and can also include the initial feature information of the fourth pixel in each image. The fourth pixel and the first pixel are different pixels in the same image.

[0123] In this embodiment, during the feature extraction stage of the first model, after obtaining the initial feature information of the first pixel in the image of the first scene from a certain perspective, the feature information of the second pixel can be obtained from the images of the first scene from other perspectives. The first pixel and the second pixel have the same semantics. The initial feature information of the second pixel is fused with the initial feature information of the first pixel to obtain the updated feature information of the first pixel, which is to say, the updated feature information of the first pixel included in the feature information of each image in the training samples is obtained. Through the aforementioned method, the updated feature information of the first pixel includes the feature information of a point in space from multiple perspectives. Thus, the feature information of each image can reflect the geometric constraints between images from different perspectives, which is beneficial to obtaining richer information and improving the accuracy of the prediction results output by the first model.

[0124] For example, the training device may randomly sample from multiple pixels included in each image to obtain multiple first pixels in each image; or, the training device may randomly sample from regions of each image that carry rich information; or, the training device may use other methods to obtain multiple first pixels from each image. The examples given here are only to demonstrate the feasibility of this solution and are not intended to limit this solution.

[0125] In the process of updating the initial feature information of any first pixel, the training device may optionally acquire the initial feature information of at least one second pixel. The first pixel and the second pixel are points in different images included in the training samples, and the first pixel and the second pixel have the same semantics. The training device fuses the initial feature information of the first pixel with the initial feature information of all second pixels to obtain the feature information of the first pixel.

[0126] For example, the first pixel is a point in the third image included in the training samples, and the second pixel is a point in the fourth image included in the training samples. The third and fourth images are different images in the training samples. It should be noted that the third image and the first image can be the same image or different images; the third image and the second image can be the same image or different images. Similarly, the fourth image and the first image can be the same image or different images; the fourth image and the second image can be the same image or different images. As long as the third and fourth images are different images in the training samples, this application does not limit the relationship between the first and second images and the third image, nor does it limit the relationship between the first and second images and the fourth image.

[0127] Regarding the specific implementation of "obtaining the initial feature information of the second pixel," for example, the training device can acquire at least one third point on a third line, the third line passing through the first pixel, and the third line also passing through the focal point of the camera capturing the third image and / or the origin of the image coordinate system corresponding to the third image.

[0128] For example, the training device may randomly sample multiple third points from the third line; or it may uniformly sample multiple third points from the third line, such that the distance between different third points is the same, etc., or it may obtain at least one third point from the third line in other ways, which will not be exhaustive here.

[0129] After determining at least one third point, the training device projects each third point onto all fourth images, obtaining multiple projection points corresponding to the at least one third point. The fourth images are images other than the third images in the training samples, and a projection point is a point on which a third point is projected. The multiple projection points include second pixels. The training device acquires a feature information set, which includes the initial feature information of each projection point among the multiple projection points.

[0130] To further understand this solution, an example of the formula used in the above steps is provided below:

[0131]

[0132] in, Representing the third line, This represents the coordinate information of the first pixel in the i-th image (i.e., an example of the third image) in the training samples. The aforementioned coordinate information is two-dimensional, K. i Represents the intrinsic parameters of the camera that captured the i-th image, [R i ,t i ] represents the extrinsic parameters of the camera that captured the i-th image.

[0133] Training equipment along the ray Sample N s The third point That is, s = 1, 2, ..., N s ; will p s The formula used to project all third points in the image to the j-th image (i.e., any fourth image) is as follows:

[0134]

[0135] in, The representative will p s The coordinate information of the projection point in the j-th image, wherein the aforementioned coordinate information is two-dimensional coordinates. represent The depth information, that is The distance K between the camera that took the j-th image and the camera that took the j-th image j Represents the intrinsic parameters of the camera that captured the j-th image, [R j ,t j ] represents the extrinsic parameters of the camera that captured the j-th image, then the camera... Projecting all the upsampled third points onto the j-th image yields multiple projection points, that is... It should be understood that the examples in Equations (1) and (2) are only for the convenience of understanding this scheme and are not intended to limit this scheme.

[0136] For a more intuitive understanding of this solution, please refer to [link / reference]. Figure 6 , Figure 6 A schematic diagram of multiple projection points provided in this application. Figure 6 Includes three sub-diagrams: (a), (b), and (c). Figure 6 The three sub-illustrative diagrams (a), (b), and (c) represent images of the same first scene from three different perspectives. Figure 6 (a) Sub-illustration represents the third image. Figure 6 Sub-illustration (b) and sub-illustration (c) are two different fourth images. Figure 6 The sub-diagram (a) shows the third point, from which four points are sampled on the third line passing through the third point. Figure 6 The sub-diagram (b) shows the projection of the aforementioned four sampled points onto... Figure 6 The four projection points obtained in sub-diagram (b) Figure 6 The sub-diagram (c) shows the projection of the aforementioned four sampled points onto... Figure 6 The four projection points obtained in sub-diagram (c) should be understood as follows: Figure 6 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0137] Regarding the specific implementation of "fusing the initial feature information of the first pixel with the initial feature information of all second pixels", in one implementation, after obtaining the feature information set, the training device can first aggregate the initial feature information of all projection points in each fourth image in the feature information set to obtain the first aggregated feature information; the training device then fuses the first aggregated feature information of all fourth images with the initial feature information of the first pixel to obtain the updated feature information of the first pixel.

[0138] To further understand this scheme, the following example illustrates the formula used in the above steps, taking the updating of the initial feature information of the first pixel using an attention mechanism:

[0139]

[0140] Among them, g j f represents the feature information after aggregating the initial feature information of multiple projection points in the j-th image (i.e., the first aggregated feature information), i Represents u in the i-th image (i.e., an example of the third image). i Initial feature information (i.e., an example of the first pixel), This represents the initial feature information of the s-th projection point in the j-th image. The feature information (position embedding) representing the position of the third point corresponding to the s-th projection point, where the position of the third point can be its three-dimensional coordinates; f i As the query matrix in the multi-head attention mechanism (MHA), As the key matrix and value matrix in MHA.

[0141]

[0142] Among them, f i ′ f represents the updated feature information of the first pixel. i The meaning can be found in the above description, N near There are N samples in total. near One image, f represents the first aggregated feature information of all fourth images in the training samples. i As the query matrix in MHA As the key matrix and value matrix in MHA; it should be understood that the examples in equations (3) and (4) are only to demonstrate the feasibility of this scheme. Other methods can also be used to fuse the "initial feature information of all projection points" with the "initial feature information of the first pixel point". The specific fusion method is not limited in the embodiments of this application.

[0143] For a more intuitive understanding of this solution, please refer to [link / reference]. Figure 7 , Figure 7 Another schematic diagram of the multiple projection points provided in this application. Figure 7 Includes three sub-diagrams: (a), (b), and (c). Figure 7 Taking a training sample consisting of three images as an example, when updating the initial feature information of a first pixel in the third image, initial feature information of multiple projection points can be obtained from each image other than the third image in the training sample (i.e., each fourth image). The initial feature information of the first pixel is used as a query matrix, and the initial feature information of all projection points in the fourth image is used as a key matrix. An attention-based feature processing model is then used to process these matrixes to obtain intermediate results. Then, the initial feature information of the first pixel is used as a query matrix, and the intermediate results corresponding to the fourth image are used as a key matrix. An attention-based feature processing model is then used to process these matrixes to obtain the updated feature information of the first pixel. It should be understood that... Figure 7 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0144] In another implementation, after the training device acquires the feature information set, it can also fuse the initial feature information of each projection point in the feature information set with the initial feature information of the first pixel to obtain the updated feature information of the first pixel.

[0145] The training device can update the initial feature information of each first pixel in the training sample in the above manner to obtain the updated feature information of each first pixel, and then obtain the feature information of each image in the training sample. The feature information of each image in the training sample can include the updated feature information of each first pixel in each image, and can also include the initial feature information of the fourth pixel in each image. The fourth pixel and the first pixel are different pixels in the same image.

[0146] In this embodiment of the application, a simple scheme for obtaining the initial feature information of the second pixel is provided, which reduces the implementation difficulty of the scheme while ensuring that the initial feature information of the second pixel can be accurately obtained.

[0147] In another implementation, step 502 may include: the training device may extract features from each image in the training samples using the first model to obtain initial feature information for each image; the feature information for each image in the training samples includes the initial feature information for each image in the training samples.

[0148] 503. Based on the feature information of each image in the training samples, feature processing is performed through the first model to obtain the prediction result output by the first model.

[0149] In this embodiment, the specific form of the "prediction result output by the first model" is similar to that of the "expected result corresponding to each training sample" mentioned above, and it also needs to be determined in conjunction with the type of task performed by the first model. For example, if the task of the first model is to generate an image of the first scene from a new perspective based on images of the first scene from multiple known perspectives, then the "training sample" may also include the new perspective, and the "prediction result output by the first model" may be the predicted image of the first scene from the aforementioned new perspective.

[0150] If the task of the first model is to generate an image of the first scene from a new perspective, the training device can perform step 503 in a variety of ways. In one implementation, the predicted image corresponding to the training sample includes the first color values ​​of multiple pixels. Step 503 may include: the training device performs feature processing on the first model based on the feature information of each image in the training sample to obtain the information generated by the first model corresponding to each third pixel. The third pixel is any one of the multiple pixels. The information corresponding to each third pixel includes multiple second color values ​​and a voxel density corresponding to each second color value.

[0151] For example, the predicted image may include multiple pixels. For each third pixel in the predicted image, before performing step 501, the training device may obtain a fourth line corresponding to each third pixel. The fourth line starts from the origin of the camera coordinate system corresponding to the predicted image and passes through the third pixel. The training device samples on the fourth line to obtain multiple sampling points corresponding to the third pixel. The training device repeats the aforementioned steps to obtain a set of sampling points corresponding to all pixels in the predicted image. The set of sampling points includes multiple sampling points corresponding to each pixel in the predicted image. The training samples may also include the aforementioned set of sampling points.

[0152] For a more intuitive understanding of this solution, please refer to [link / reference]. Figure 8 , Figure 8 A schematic diagram of the fourth line provided in an embodiment of this application, such as... Figure 8 As shown, the origin of the camera coordinate system corresponding to the predicted image is 01, and the X-axis of this camera coordinate system is... Figure 4In the context of camera Xc, the Y-axis of the camera coordinate system is... Figure 4 In the Yc coordinate system of the camera, the Z-axis is... Figure 4 In the camera Zc, taking the acquisition of the first color value of pixel b in the predicted image as an example, a ray can be emitted from the origin O1 of the camera coordinate system to pixel b, and this ray passes through pixel b to obtain the fourth line corresponding to pixel b; sampling can be performed on the fourth line corresponding to pixel b to obtain multiple sampling points corresponding to pixel b. It should be understood that... Figure 8 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0153] The training device inputs training samples into the first model and can obtain the information generated by the first model corresponding to each third pixel. The information corresponding to each third pixel includes the second color value and voxel density of multiple sampling points (hereinafter referred to as "target sampling points" for ease of description) corresponding to each third pixel.

[0154] The training device normalizes the voxel density corresponding to each second color value to obtain the weight of each second color value; that is, it normalizes the voxel density corresponding to each target sampling point to obtain the weight of the second color value of that target sampling point. The training device then performs a weighted sum of the weights of the second color values ​​of the multiple target sampling points corresponding to the third pixel, and the sum of these weights to obtain the first color value of the third pixel. The training device repeats the aforementioned steps multiple times to obtain the first color value corresponding to each pixel in the prediction image, thus determining the prediction image.

[0155] To further understand this scheme, the following is an example of the formula used to calculate the first color value of a third pixel:

[0156]

[0157] in, Let represent the first color value of a third pixel in the predicted image. Here, we assume the origin of the camera coordinate system corresponding to the predicted image is taken as the starting point, and the fourth line passing through the third pixel is ray r. i σ represents the second color value of the i-th sampling point on ray r (that is, the i-th sampling point corresponding to the third pixel). i The voxel density at the i-th sampling point on ray r is exp(σ i ) represents the relationship between σ and σ. i The weight of the i-th sampling point on ray r after normalization should be understood as an example only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0158] In this embodiment of the application, the above-described scheme provides a method for obtaining the color values ​​of pixels in a predicted image, and the above scheme is simple and convenient.

[0159] In one implementation, step 503 may include: the training samples include only images of the first scene from multiple known viewpoints and the new viewpoint; the training device performs feature processing on each image in the training samples using the first model to obtain the prediction result output by the first model, and the prediction result is the image of the first scene from the new viewpoint.

[0160] For example, if the task of the first model is to classify the first scene based on images of the first scene from multiple perspectives, then the "prediction result corresponding to the training samples" can be represented as the predicted category of the first scene. For another example, if the task of the first model is to detect objects in the first scene based on images of the first scene from multiple perspectives, then the "prediction result corresponding to the training samples" can be represented as the location information of at least one detected object in the first scene.

[0161] For example, if the task of the first machine learning model is to perform semantic segmentation on images of the first scene from multiple perspectives, then the "detection results corresponding to the training samples" can be represented as the detection category of each pixel in each of the aforementioned multiple images, etc. It should be understood that the examples here are only for the convenience of understanding this scheme and are not intended to limit this scheme.

[0162] 504. The first model is trained based on the feature information of each image in the training samples and the first loss term. The goal of training with the first loss term is to improve the similarity between the first feature information and the second feature information. The first feature information includes the feature information of the first point in the first image, and the second feature information includes the feature information of the second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0163] In this embodiment, the training device can generate the value of a loss function based on the feature information of each image in the training samples. The loss function includes a first loss term. Based on the value of the loss function, the training device updates the weight parameters of the first model using backpropagation to complete one training of the first model.

[0164] The first loss term indicates the similarity between the first feature information and the second feature information. The purpose of training with the first loss term includes improving the similarity between the first feature information and the second feature information. The first feature information includes the feature information of a first point in the first image, and the second feature information includes the feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. In other words, projecting the same point in the three-dimensional first scene onto images from different perspectives can yield the first point in the first image and the second point in the second image, respectively. That is, the purpose of training with the first loss term includes improving the similarity between the feature information of points with the same semantics in images from different perspectives.

[0165] To further understand this scheme, for example, the training device randomly samples N from the i-th image included in the training samples. c The first point, the aforementioned multiple first points are expressed as follows: N c For integers greater than or equal to 1, p = (p, q) is For any first point p in the i-th image, we specify q in the j-th image of the training samples. + This refers to the second point (i.e., a point semantically similar to the first point). The goal is to obtain multiple second points q corresponding to each first point p in the i-th image. + The training device can first obtain the coordinates of p3 in three-dimensional space based on the coordinates of the first point p. p is the point obtained by projecting p3 onto the i-th image. The formula for calculating the three-dimensional coordinate information of p3 can be as follows:

[0166]

[0167] in, These are the homogeneous coordinates of p, δ p K represents the distance between p3 and the camera that captured the i-th image. i Represents the intrinsic parameters of the camera that captured the i-th image, [R i ,t i ] represents the external parameters of the camera that takes the i-th image. It should be understood that the examples in Equation (6) are only for the convenience of understanding this scheme and are not intended to limit this scheme.

[0168] The training device projects p3 in three-dimensional space onto the j-th image in the training samples, obtaining the projection point q of p3 in the j-th image. + That is, we obtain a second point q in the j-th image that corresponds to the first point p. + The training device repeats the aforementioned operation to obtain the projection point q of p3 in each image other than the i-th image in the training samples.+ Thus, multiple second points q corresponding to the first point p were obtained. + .

[0169] The training device determines multiple second points q corresponding to the first point p in the i-th image. + Then, the feature information of the first point p and the feature information of each second point q can be obtained from the feature information of each image included in the training samples. + The feature information is used to calculate the feature information of the first point p and each second point q. + The similarity between the feature information; the training device repeatedly performs the above operation to obtain the similarity between the feature information of each first point in the i-th image and the feature information of the second point in other images in the training sample, and then obtains the similarity between the feature information of each first point in each image in the training sample and the feature information of the second point in other images in the training sample, thereby calculating the value of the first loss term.

[0170] It should be noted that the above examples are only to demonstrate the feasibility of this solution. The training device may also use other methods to calculate the value of the first loss term, and the embodiments of this application do not limit this.

[0171] Optionally, during the training of the first model using the loss function, step 504 may include: the training device training the first model based on the feature information of each image in the training samples and the loss function. The loss function includes a first loss term and a second loss term. The second loss term indicates the similarity between the first feature information and the third feature information. The objective of training using the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of a third point in the second image, where the third point is different from the second point.

[0172] For example, the training device can determine all third points corresponding to a first point in the i-th image included in the training samples, and calculate the similarity between the feature information of the aforementioned first point in the i-th image and the feature information of all third points; the training device repeats the aforementioned operation to obtain the similarity between the feature information of each first point in the i-th image of the training samples and the feature information of all third points. The training device repeats the aforementioned operation to obtain the similarity between the feature information of each first point in each image of the training samples and the feature information of all third points, thereby obtaining the value of the second loss term.

[0173] In one scenario, the third point in the second image can be any point in the second image other than the second point. In another scenario, both the second and third points in the second image lie on the first line. Projecting the second line onto the second image yields the first line, which passes through the first point and also through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image. It should be noted that the concept of the "second line" is similar to that of the "third line," and can be understood by referring to the description; it will not be elaborated upon here. Alternatively, the training device can also determine the third point using other methods, which will not be exhaustively listed here.

[0174] Optionally, if both the second point and the third point in the second image are located on the first line, the specific implementation process for determining all third points corresponding to a given first point is as follows: For example, in one implementation, the training device can project the second line onto the j-th image (i.e., an example of any second image) to obtain the first line, and determine a second point q corresponding to the first point p in the j-th image. + Then, from the first line and located at the second point q + In addition, multiple third points are sampled to obtain the j-th image. Among them, the first line and located at the second point q + All points other than the one mentioned above can be considered as third points. N represents the set of multiple third points in the j-th image. neg This represents the number of third points included in the j-th image of the training sample. By repeatedly performing the aforementioned steps, the training device can determine multiple third points in each second image of the training sample, thereby obtaining all the third points corresponding to the first point p.

[0175] In another implementation, after determining the second line, the training device can determine multiple sampling points from the second line, and project these multiple sampling points onto the j-th image to obtain multiple projection points. If these multiple projection points include the second point q... + Then the second point q among the aforementioned multiple projection points will be... + All projection points other than those in the j-th image are defined as all third points included in the j-th image; if these multiple projection points do not include the second point q... + Then, all the projection points among the aforementioned multiple projection points are determined as all the third points included in the j-th image. The training device repeats the aforementioned steps to determine multiple third points in each second image of the training sample, thereby obtaining all the third points corresponding to the first point p.

[0176] The training device can also obtain all the third points corresponding to each first point in other ways. The example here is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0177] For a more intuitive understanding of this solution, please refer to [link / reference]. Figure 9 , Figure 9 A schematic diagram illustrating the first, second, and third points provided in the embodiments of this application, as shown below. Figure 9 As shown, in the second image, a second point corresponding to the first point and multiple third points are located on the same line. It should be understood that... Figure 9 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0178] In this embodiment, after determining the first point, a second line can be obtained. Projecting the second line onto images from other perspectives (i.e., the second image) can determine the first line in the second image. Points with the same semantics as the first point are generally located on the first line. Sampling a third point from the first line helps the first model learn which points in images from other perspectives are consistent with the points in the first scene corresponding to the first point, and which points in images from other perspectives are inconsistent with the points in the first scene corresponding to the first point. This helps to further improve the accuracy of the prediction results output by the first model.

[0179] To further understand this scheme, an example of the loss function, which includes a first loss term and a second loss term, is presented here:

[0180]

[0181]

[0182] Where p′ represents the feature information of the first point p in the feature information of the i-th image included in the training samples, and q′ + The second point q in the feature information of the j-th image included in the training samples + Feature information, q′ - The third point q in the feature information of the j-th image included in the training samples - The feature information, τ represents the learnable scalar parameter, λ q- Representative assigned to each q - The weights, exp(p′·q′) + / τ) represents the feature information of the first point p and the second point q. + The similarity between feature information, ∑ q- λ q- exp(p′·q′ - / τ) represents the feature information of the first point p and multiple third points q. - The similarity between feature information, The first value represents a first value of the loss function corresponding to a first point p in the i-th image included in the training samples and the j-th image (i.e., any image other than the i-th image in the training samples); the training device calculates the first value between a first point p in the i-th image and each other image in the training samples (i.e., each j-th image), and averages the multiple first values ​​to obtain a second value of the loss function corresponding to a first point p in the i-th image included in the training samples and all other images.

[0183] The training device calculates N values ​​associated with the i-th image. c For each first point p in the set of first points p, calculate the second value corresponding to N. c The average of the second values ​​yields L. contrast ,L contrast The third value of the loss function corresponding to the i-th image is represented by the third value of the training device for each image in the training samples. The average of all the third values ​​is used to obtain the value of the loss function. It should be understood that the examples in Equations (7) and (8) are only for the convenience of understanding this scheme and are not intended to limit this scheme.

[0184] Because a first point p and multiple third points q in the j-th image. - Correspondingly, different third points q - The same weight λ can be used q- Different weights λ can also be used. q- Alternatively, it can be based on the second point q. + and each third point q - The distance between each third point q is used to calculate the distance between them. - weight λ q- , each third point q - With the second point q + The greater the distance between them, the more important the third point q becomes. - weight λ q- It can be larger. For example, the following discloses the calculation of the weight λ. q- An example of a formula:

[0185]

[0186] Where τ′ is a scalar parameter, the meaning of the other letters in equation (9) can be found in the descriptions of equations (7) and (8) above, and will not be repeated here. It should be understood that the examples in equation (9) are only for the convenience of understanding this scheme and are not intended to limit this scheme.

[0187] For a more intuitive understanding of this solution, please refer to [link / reference]. Figure 10 , Figure 10 This is a schematic diagram illustrating the calculation process of the loss function provided in an embodiment of this application. Figure 10Taking the calculation of a first value of the loss function corresponding to a first point in a first image and a second image as an example, after obtaining the feature information of the first point in the first image, the feature information of the second point in the second image, and the feature information of the third point, a first similarity can be calculated between the feature information of the first point and the feature information of the second point. A second similarity can also be calculated between the feature information of the first point and the feature information of multiple third points. Furthermore, the distance between the feature information of the second point and the feature information of multiple third points can be calculated. Based on the aforementioned first similarity, second similarity, and distance, a first value of the loss function corresponding to a first point in the first image and a second image is generated. For an understanding of the meaning of "first value," please refer to the above description. Figure 10 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0188] In this embodiment, since the first point in the first image and the second point in the second image correspond to the same point in the first scene, and the third point is different from the second point, that is, the third point and the first point correspond to different points in the first scene, the purpose of training with the second loss term includes reducing the similarity between the first feature information of the first point and the third feature information of the third point. That is, not only will the feature information of the same point in the first scene be more similar in images from different perspectives, but the feature information of different points in the first scene will be less similar in images from different perspectives. Therefore, the feature information obtained by the aforementioned first model can reflect more connections between images from different perspectives, which is beneficial to further enhance the geometric perception ability of the model, thereby improving the accuracy of the prediction results output by the first model.

[0189] Optionally, the loss function in step 504 may also include a third loss term, which indicates the similarity between the expected result corresponding to the training sample and the prediction result output by the first model. The purpose of training with the third loss term includes improving the similarity between the expected result corresponding to the training sample and the prediction result output by the first model.

[0190] For example, if the task of the first model is to generate an image of the first scene from a new perspective, then the third loss term indicates the similarity between the predicted image and the expected image of the first scene from the new perspective. Step 504 may include: the training device training the first model based on the feature information of each image in the training samples, the predicted image, and the loss function, wherein the loss function includes the first loss term and the third loss term.

[0191] In training the first model, to ensure its output closely approximates the desired value, the predicted value is compared to the expected value. The weight vectors of each layer are updated based on the difference (usually pre-configured before the initial update). For example, if the predicted value is too high, the weight vectors are adjusted to predict a lower value. This adjustment continues until the first model predicts the desired target value or a value very close to it. Therefore, it's necessary to pre-define "how to compare the difference between the predicted and target values," which is the third loss term. These are crucial equations used to measure the difference between the predicted and target values. For example, a higher loss function indicates a greater difference, making the training of a deep neural network a process of minimizing this loss.

[0192] This application provides a specific application scenario for the solution, which improves the integration of the solution with actual application scenarios and reduces the implementation difficulty of the solution.

[0193] Optionally, step 504 may include: the training device training the first model based on the feature information of each image in the training samples, the predicted image, and the loss function, wherein the loss function includes a first loss term, a second loss term, and a third loss term.

[0194] The training device repeats steps 501 to 504 to iteratively train the first model until the convergence condition of the loss function is met, thus obtaining the trained first model.

[0195] II. Reasoning Stage

[0196] In this embodiment, the inference phase describes how the execution device 230 uses the first model 201 to output a prediction result corresponding to the data to be processed. For details, please refer to [link to relevant documentation]. Figure 11 , Figure 11 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. The image processing method provided in an embodiment of this application may include:

[0197] 1101. Obtain the data to be processed. The data to be processed includes multiple images of the scene from multiple perspectives.

[0198] 1102. Input the data to be processed into the first model, and perform feature extraction through the first model to obtain the feature information of each image in the data to be processed.

[0199] Optionally, in this embodiment of the application, the feature information of each image in the data to be processed includes the feature information of multiple points of each image in the data to be processed, and the multiple points include a first pixel. Step 1102 may include: the execution device obtaining the initial feature information of a second pixel, where the first pixel and the second pixel are points in different images in the data packet to be processed, and the first pixel and the second pixel have the same semantics; and fusing the initial feature information of the first pixel and the initial feature information of the second pixel to obtain the feature information of the first pixel.

[0200] 1103. Based on the feature information of each image in the data to be processed, feature processing is performed through the first model to obtain the prediction result output by the first model.

[0201] Optionally, in this embodiment of the application, the first model is trained based on training samples and a first loss term. The multiple images in the training samples include images of the first scene from multiple perspectives. The multiple images in the training samples include a first image and a second image. The first loss term indicates the similarity between first feature information and second feature information. The first feature information includes feature information of a first point in the first image. The second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0202] Optionally, the first model is trained based on training samples, a first loss term, and a second loss term. The objective of training using the second loss term includes reducing the similarity between the first and third feature information.

[0203] In this embodiment of the application, the first model in steps 1101 to 1103 is obtained through the above... Figure 5 The training method of the model described in the corresponding embodiment is used to obtain the specific implementation of steps 1101 to 1103 of the execution device. The meanings of the terms in steps 1101 to 1103 can be found in the above description. Figure 5 The difference is that the description in the corresponding embodiment is as follows: Figure 5 In the corresponding embodiment, "training samples" are replaced with Figure 11 The “data to be processed” in the corresponding embodiments will not be described in detail here.

[0204] To more intuitively understand the beneficial effects of this solution, the following description, based on experimental data, further illustrates the advantages of the embodiments of this application. The experiment uses the task of generating images of the first scene from a new perspective as an example, and selects 3D-FRONT and ScanNet as the training and testing datasets. Three metrics are used to evaluate the model: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS). Higher PSNR and SSIM indicate better prediction results, while lower LPIPS also indicate better prediction results. The experimental results are shown in Table 1 below.

[0205] Pixe1NeRF 20.19(22.44) 0.736(0.774) 0.511(0.450) MVSNeRF 22.90(24.90) 0.793(0.824) 0.408(0.357) Neuray 22.75(25.22) 0.785(0.838) 0.369(0.325) This application 24.81(25.58) 0.831(0.847) 0.333(0.320)

[0206] Table 1

[0207] Among them, Pixe1NeRF, MVSNeRF, and Neuray are three existing models. The experimental data in Table 1 were obtained by training the aforementioned three models using existing methods. The data outside the parentheses in Table 1 represents models trained with simulation data and tested with real data; the data inside the parentheses represents models trained with real data and tested with real data. Referring to the data in Table 1, it can be seen that the model obtained based on the scheme provided in this application has the best prediction results.

[0208] In Figure 1 to Figure 11 Based on the corresponding embodiments, in order to better implement the above-described solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 12 , Figure 12 This is a schematic diagram of a model training device provided in an embodiment of this application. The model training device 1200 includes: an acquisition module 1201, used to acquire training samples, wherein multiple images in the training samples include images of a first scene from multiple viewpoints; a processing module 1202, used to input the training samples into a first model, and perform feature extraction through the first model to obtain feature information of each image in the training samples; and a training module 1203, used to train the first model based on the feature information of each image in the training samples and a first loss term.

[0209] The multiple images include a first image and a second image. The goal of training using a first loss term is to improve the similarity between the first feature information and the second feature information. The first feature information includes the feature information of a first point in the first image, and the second feature information includes the feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0210] In one possible design, the training module 1203 is specifically used to train the first model based on the feature information of each image in the training samples, a first loss term, and a second loss term. The objective of training using the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of a third point in the second image, which is different from the second point.

[0211] In one possible design, both the second and third points are located on the first line. The second line is projected onto the second image to obtain the first line. The second line passes through the first point and also passes through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image.

[0212] In one possible design, the feature information of each image in the training sample includes the updated feature information of multiple first pixels in each image in the training sample; the processing module 1202 is specifically used to obtain the initial feature information of the second pixel, and fuse the initial feature information of the first pixel with the initial feature information of the second pixel to obtain the updated feature information of the first pixel, wherein the first pixel and the second pixel are points in different images in multiple images, and the first pixel and the second pixel have the same semantics.

[0213] In one possible design, the first pixel is contained in the third image of the training samples. The processing module 1202 includes: an acquisition submodule for acquiring at least one third point on a third line, the third line passing through the first pixel, and the third line also passing through the focal point of the camera capturing the third image and / or the origin of the image coordinate system corresponding to the third image; the acquisition submodule is further used to acquire a feature information set, the feature information set including the initial feature information of each of a plurality of projection points, the projection points being points that project the third point onto a fourth image, the fourth image being an image other than the third image in the training samples, and the plurality of projection points including the second pixel; and a fusion submodule for fusing the initial feature information of the first pixel with the feature information set.

[0214] In one possible design, the training samples also include a new perspective. The processing module 1202 is further used to perform feature processing on the first model based on the feature information of each image in the training samples to obtain a predicted image of the first scene under the new perspective. The training module 1203 is specifically used to train the first model based on the feature information of each image in the training samples, the predicted image, the first loss term, and the third loss term. The third loss term indicates the similarity between the predicted image and the expected image of the first scene under the new perspective.

[0215] In one possible design, the predicted image includes first color values ​​of multiple pixels. The processing module 1202 includes: a processing submodule for performing feature processing through a first model to obtain information corresponding to a third pixel generated by the first model, wherein the third pixel is any one of the multiple pixels, and the information corresponding to the third pixel includes multiple second color values ​​and a voxel density corresponding to each second color value; a normalization submodule for normalizing the voxel density corresponding to each second color value to obtain the weight of each second color value; and a weighted summation submodule for performing weighted summation of the multiple second color values ​​according to the weight of each second color value to obtain the first color value of the third pixel.

[0216] It should be noted that the information interaction and execution process between the modules / units in the model training device 1200 are different from those in this application. Figures 3 to 11 The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0217] Please see Figure 13 , Figure 13 This is a schematic diagram of an image processing apparatus provided in an embodiment of this application. The image processing apparatus 1300 includes: an acquisition module 1301, used to acquire data to be processed, wherein multiple images in the data to be processed include images of a scene from multiple viewpoints; a processing module 1302, used to input the data to be processed into a first model, and perform feature extraction through the first model to obtain feature information of each image in the data to be processed; the processing module 1302 is further used to perform feature processing through the first model based on the feature information of each image in the data to be processed to obtain a prediction result output by the first model;

[0218] The first model is trained based on training samples and a first loss term. The training samples include multiple images of the first scene from multiple perspectives. The training samples include a first image and a second image. The first loss term indicates the similarity between first feature information and second feature information. The first feature information includes feature information of a first point in the first image. The second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0219] In one possible design, the first model is trained based on training samples, a first loss term, and a second loss term. The goal of training with the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of the first pixel in the second image, and the first pixel and the second pixel are different.

[0220] In one possible design, the feature information of each image in the data to be processed includes the feature information of multiple first pixels in each image in the data to be processed; the processing module 1302 is specifically used to obtain the initial feature information of the second pixel, and fuse the initial feature information of the first pixel with the initial feature information of the second pixel to obtain the feature information of the first pixel, wherein the first pixel and the second pixel are points in different images in the data packet to be processed, and the first pixel and the second pixel have the same semantics.

[0221] It should be noted that the information interaction and execution process between the modules / units in the image processing apparatus 1300 are different from those in this application. Figures 3 to 11 The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0222] Please see Figure 14 , Figure 14 This is a schematic diagram of an image processing apparatus provided in an embodiment of this application. The image processing apparatus 1400 includes: an acquisition module 1401, used to acquire data to be processed, wherein multiple images in the data to be processed include images of a scene from multiple viewpoints; a processing module 1402, used to input the data to be processed into a first model, and perform feature extraction through the first model to obtain feature information of each image in the data to be processed; the processing module is further used to perform feature processing through the first model based on the feature information of each image in the data to be processed to obtain a prediction result output by the first model;

[0223] The feature information of each image in the data to be processed includes the feature information of multiple first pixels in each image in the data to be processed; the processing module 1402 is specifically used to obtain the initial feature information of the second pixel, and fuse the initial feature information of the first pixel with the initial feature information of the second pixel to obtain the feature information of the first pixel, wherein the first pixel and the second pixel are points in different images in the data to be processed, and the first pixel and the second pixel have the same semantics.

[0224] In one possible design, the first model is trained based on training samples and a first loss term. The training samples include multiple images of the first scene from multiple perspectives. The training samples include a first image and a second image. The first loss term indicates the similarity between first feature information and second feature information. The first feature information includes feature information of a first point in the first image. The second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene.

[0225] It should be noted that the information interaction and execution process between the modules / units in the image processing apparatus 1400 are different from those in this application. Figures 3 to 11 The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0226] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 15 , Figure 15 This is a schematic diagram of the structure of an execution device provided in an embodiment of this application. Specifically, the execution device 1500 includes: a receiver 1501, a transmitter 1502, a processor 1503, and a memory 1504 (wherein the execution device 1500 may have one or more processors 1503). Figure 15 (Taking a processor as an example), processor 1503 may include application processor 15031 and communication processor 15032. In some embodiments of this application, receiver 1501, transmitter 1502, processor 1503 and memory 1504 may be connected via a bus or other means.

[0227] Memory 1504 may include read-only memory and random access memory, and provides instructions and data to processor 1503. A portion of memory 1504 may also include non-volatile random access memory (NVRAM). Memory 1504 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0228] Processor 1503 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0229] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 1503. Processor 1503 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 1503 or by instructions in software form. Processor 1503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1503 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1504. Processor 1503 reads the information in memory 1504 and, in conjunction with its hardware, completes the steps of the above method.

[0230] Receiver 1501 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1502 can be used to output digital or character information through the first interface; transmitter 1502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1502 may also include a display device such as a display screen.

[0231] In this embodiment of the application, the application processor 15031 in the processor 1503 is used to execute... Figures 3 to 11 The image processing method executed by the execution device in the corresponding embodiment. It should be noted that the specific manner in which the application processor 15031 executes the above steps differs from that in this application. Figures 3 to 11 The various method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figures 3 to 11 The corresponding method embodiments are the same, and for details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0232] This application also provides a training device; please refer to [link / reference]. Figure 16 , Figure 16 This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 1600 is implemented by one or more servers. The training device 1600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1622 (e.g., one or more processors) and memory 1632, and one or more storage media 1630 (e.g., one or more mass storage devices) for storing application programs 1642 or data 1644. The memory 1632 and storage media 1630 can be temporary or persistent storage. The program stored in the storage media 1630 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1622 may be configured to communicate with the storage media 1630 and execute the series of instruction operations in the storage media 1630 on the training device 1600.

[0233] The training device 1600 may also include one or more power supplies 1626, one or more wired or wireless network interfaces 1650, one or more input / output interfaces 1658, and / or one or more operating systems 1641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0234] In this embodiment, the central processing unit 1622 is used to execute... Figures 3 to 11 The training method for the model executed by the training device in the corresponding embodiment. It should be noted that the specific manner in which the central processing unit 1622 executes the above steps differs from that in this application. Figure 12 The various method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figure 12 The corresponding method embodiments are the same, and for details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0235] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 3 to 11 The method described in the illustrated embodiment executes steps performed by the execution device, or causes the computer to perform the steps as described above. Figures 3 to 11 The steps performed by the training device in the method described in the illustrated embodiment.

[0236] This application embodiment also provides a computer-readable storage medium storing a program for performing signal processing, which, when run on a computer, causes the computer to perform the aforementioned actions. Figures 3 to 11 The method described in the illustrated embodiment executes steps performed by the execution device, or causes the computer to perform the steps as described above. Figures 3 to 11 The steps performed by the training device in the method described in the illustrated embodiment.

[0237] The execution device, training device, model training apparatus, or image processing apparatus provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuitry. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip to perform the aforementioned operations. Figures 3 to 11 The image processing method described in the illustrated embodiment, or, to cause the chip to perform the above... Figures 3 to 11 The training method of the model described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.

[0238] For details, please refer to Figure 17 , Figure 17This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 170. The NPU 170 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 170, which is controlled by the controller 1704 to extract matrix data from the memory and perform multiplication operations.

[0239] In some implementations, the arithmetic circuit 1703 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1703 is a two-dimensional pulsating array. The arithmetic circuit 1703 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1703 is a general-purpose matrix processor.

[0240] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1702 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1701 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1708.

[0241] Unified memory 1706 is used to store input and output data. Weight data is directly transferred to weight memory 1702 via Direct Memory Access Controller (DMAC) 1705. Input data is also transferred to unified memory 1706 via DMAC.

[0242] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1709.

[0243] The Bus Interface Unit (BIU) 1710 is used by the instruction fetch memory 1709 to fetch instructions from external memory, and also by the memory access controller 1705 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0244] The DMAC is mainly used to move input data from external memory DDR to unified memory 1706, or to weight data to weight memory 1702, or to input data to input memory 1701.

[0245] The vector computation unit 1707 includes multiple processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0246] In some implementations, the vector computation unit 1707 can store the processed output vector in the unified memory 1706. For example, the vector computation unit 1707 can apply linear and / or nonlinear functions to the output of the computation circuit 1703, such as performing linear interpolation on feature planes extracted from convolutional layers, or accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1707 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1703, for example, for use in subsequent layers of the neural network.

[0247] The instruction fetch buffer 1709 connected to the controller 1704 is used to store the instructions used by the controller 1704.

[0248] Unified memory 1706, input memory 1701, weighted memory 1702, and instruction fetch memory 1709 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0249] In the first model, the operations of each layer can be performed by the operation circuit 1703 or the vector calculation unit 1707.

[0250] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0251] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0252] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0253] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0254] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Acquire training samples, wherein the training samples include multiple images of the first scene from multiple viewpoints; The training samples are input into the first model, and feature extraction is performed through the first model to obtain the feature information of each image in the training samples; The first model is trained based on the feature information of each image in the training samples and the first loss term. The plurality of images include a first image and a second image. The goal of training using the first loss term includes improving the similarity between the first feature information and the second feature information. The first feature information includes the feature information of a first point in the first image, and the second feature information includes the feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. The same point is a three-dimensional spatial point. The first point is the projection of the three-dimensional spatial point onto the first image, and the second point is the projection of the three-dimensional spatial point onto the second image. The step of training the first model based on the feature information of each image in the training samples and the first loss term includes: training the first model based on the feature information of each image in the training samples, the first loss term, and the second loss term. The goal of training using the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of a third point in the second image. The third point is different from the second point. Both the second point and the third point are located on a first line. The second line is projected onto the second image to obtain the first line. The second line passes through the first point and passes through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image.

2. The method according to claim 1, characterized in that, The feature information of each image in the training sample includes the updated feature information of multiple first pixels in each image in the training sample. The feature extraction process using the first model to obtain the feature information of each image in the training sample includes: The initial feature information of the second pixel is obtained. The first pixel and the second pixel are points in different images among the plurality of images. The first pixel and the second pixel have the same semantics. The same semantics means that the first pixel and the second pixel are projections of the same three-dimensional space point on their respective images. The initial feature information of the first pixel is fused with the initial feature information of the second pixel to obtain the updated feature information of the first pixel.

3. The method according to claim 2, characterized in that, The first pixel is contained in the third image of the training sample, and obtaining the initial feature information of the second pixel includes: Obtain at least one third point on the third line, the third line passing through the first pixel, and the third line also passing through the focal point of the camera that captured the third image and / or the origin of the image coordinate system corresponding to the third image; A feature information set is obtained, the feature information set includes the initial feature information of each projection point among multiple projection points, the projection point is the point on which the third point is projected onto the fourth image, the fourth image is an image other than the third image in the training sample, and the multiple projection points include the second pixel point; The step of fusing the initial feature information of the first pixel with the initial feature information of the second pixel includes: The initial feature information of the first pixel is fused with the feature information set.

4. The method according to claim 1, characterized in that, The training samples also include new perspectives. After extracting features from each image in the training samples using the first model, the method further includes: Based on the feature information of each image in the training samples, feature processing is performed by the first model to obtain the predicted image of the first scene under the new perspective. The step of training the first model based on the feature information of each image in the training samples and the first loss term includes: The first model is trained based on the feature information of each image in the training samples, the predicted image, the first loss term, and the third loss term, wherein the third loss term indicates the similarity between the predicted image and the expected image of the first scene from the new perspective.

5. The method according to claim 4, characterized in that, The predicted image includes the first color values ​​of multiple pixels. The step of performing feature processing through the first model to obtain the predicted image of the first scene from the new perspective includes: By performing feature processing through the first model, information corresponding to the third pixel point generated by the first model is obtained. The third pixel point is any one of the plurality of pixels. The information corresponding to the third pixel point includes a plurality of second color values ​​and a voxel density corresponding to each second color value. The voxel density corresponding to each second color value is normalized to obtain the weight of each second color value; The first color value of the third pixel is obtained by weighted summation of the plurality of second color values ​​according to the weight of each second color value.

6. An image processing method, characterized in that, The method includes: Acquire data to be processed, wherein the data to be processed includes multiple images of the scene from multiple perspectives; The data to be processed is input into the first model, and feature extraction is performed through the first model to obtain the feature information of each image in the data to be processed. Based on the feature information of each image in the data to be processed, feature processing is performed through the first model to obtain the prediction result output by the first model; The first model is trained based on training samples and a first loss term. The training samples include multiple images of the first scene from multiple perspectives. The training samples include a first image and a second image. The first loss term indicates the similarity between first feature information and second feature information. The first feature information includes feature information of a first point in the first image. The second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. The same point is a three-dimensional spatial point. The first point is the projection of the three-dimensional spatial point onto the first image, and the second point is the projection of the three-dimensional spatial point onto the second image. The first model is trained based on training samples and a first loss term, including: training the first model according to the feature information of each image in the training samples, the first loss term, and the second loss term. The goal of training with the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of a third point in the second image. The third point is different from the second point. Both the second point and the third point are located on a first line. The second line is projected onto the second image to obtain the first line. The second line passes through the first point and passes through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image.

7. The method according to claim 6, characterized in that, The feature information of each image in the data to be processed includes the feature information of multiple first pixels of each image in the data to be processed. The step of extracting features through the first model to obtain the feature information of each image in the data to be processed includes: The initial feature information of the second pixel is obtained. The first pixel and the second pixel are points in different images in the data packet to be processed. The first pixel and the second pixel have the same semantics. The same semantics means that the first pixel and the second pixel are projections of the same three-dimensional space point on their respective images. The initial feature information of the first pixel is fused with the initial feature information of the second pixel to obtain the feature information of the first pixel.

8. An image processing method, characterized in that, The method includes: Acquire data to be processed, wherein the data to be processed includes multiple images of the scene from multiple perspectives; The data to be processed is input into the first model, and feature extraction is performed through the first model to obtain the feature information of each image in the data to be processed. Based on the feature information of each image in the data to be processed, feature processing is performed through the first model to obtain the prediction result output by the first model; The feature information of each image in the data to be processed includes the feature information of multiple first pixels of each image in the data to be processed. The step of extracting features using the first model to obtain the feature information of each image in the data to be processed includes: The initial feature information of the second pixel is obtained. The first pixel and the second pixel are points in different images in the data packet to be processed. The first pixel and the second pixel have the same semantics. The same semantics means that the first pixel and the second pixel are projections of the same three-dimensional space point on their respective images. The initial feature information of the first pixel is fused with the initial feature information of the second pixel to obtain the feature information of the first pixel; The first model is trained based on training samples, a first loss term, and a second loss term. The training samples include multiple images of a first scene from multiple viewpoints, including a first image and a second image. The first loss term indicates the similarity between first and second feature information. The first feature information includes feature information of a first point in the first image, and the second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. The goal of training with the second loss term is to reduce the similarity between the first and third feature information. The third feature information includes feature information of a third point in the second image, which is different from the second point. Both the second point and the third point are located on a first line. The second line is projected onto the second image to obtain the first line. The second line passes through the first point and passes through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image.

9. A training device for a model, characterized in that, The training device for the model includes: The acquisition module is used to acquire training samples, wherein the training samples include multiple images of the first scene from multiple viewpoints; The processing module is used to input the training samples into the first model, and perform feature extraction through the first model to obtain the feature information of each image in the training samples; The training module is used to train the first model based on the feature information of each image in the training samples and the first loss term; The plurality of images include a first image and a second image. The goal of training using the first loss term includes improving the similarity between the first feature information and the second feature information. The first feature information includes the feature information of a first point in the first image, and the second feature information includes the feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. The same point is a three-dimensional spatial point. The first point is the projection of the three-dimensional spatial point onto the first image, and the second point is the projection of the three-dimensional spatial point onto the second image. The training module is specifically used to train the first model based on the feature information of each image in the training samples, a first loss term, and a second loss term. The goal of training using the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of a third point in the second image. The third point is different from the second point. Both the second point and the third point are located on a first line. The second line is projected onto the second image to obtain the first line. The second line passes through the first point and also passes through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image.

10. The apparatus according to claim 9, characterized in that, The feature information of each image in the training sample includes the updated feature information of multiple first pixels of each image in the training sample; The processing module is specifically used to obtain the initial feature information of the second pixel, and fuse the initial feature information of the first pixel with the initial feature information of the second pixel to obtain the updated feature information of the first pixel. The first pixel and the second pixel are points in different images among the plurality of images. The first pixel and the second pixel have the same semantics. The same semantics means that the first pixel and the second pixel are projections of the same three-dimensional space point on their respective images.

11. The apparatus according to claim 10, characterized in that, The first pixel is contained in the third image of the training samples. The processing module includes: The acquisition submodule is used to acquire at least one third point on the third line, the third line passing through the first pixel point, and the third line also passing through the focal point of the camera that captures the third image and / or the origin of the image coordinate system corresponding to the third image; The acquisition submodule is further configured to acquire a feature information set, the feature information set including the initial feature information of each projection point among multiple projection points, the projection point being the point on which the third point is projected onto the fourth image, the fourth image being an image other than the third image in the training samples, and the multiple projection points including the second pixel point; The fusion submodule is used to fuse the initial feature information of the first pixel with the feature information set.

12. The apparatus according to claim 9, characterized in that, The training samples also include a new perspective. The processing module is further configured to perform feature processing on each image in the training samples using the first model to obtain a predicted image of the first scene under the new perspective. The training module is specifically used to train the first model based on the feature information of each image in the training samples, the predicted image, the first loss term, and the third loss term, wherein the third loss term indicates the similarity between the predicted image and the expected image of the first scene under the new perspective.

13. The apparatus according to claim 12, characterized in that, The predicted image includes first color values ​​for multiple pixels, and the processing module includes: The processing submodule is used to perform feature processing through the first model to obtain information generated by the first model corresponding to the third pixel point. The third pixel point is any one of the plurality of pixels. The information corresponding to the third pixel point includes a plurality of second color values ​​and a voxel density corresponding to each second color value. The normalization submodule is used to normalize the voxel density corresponding to each second color value to obtain the weight of each second color value; The weighted summation submodule is used to perform a weighted summation of the plurality of second color values ​​according to the weight of each second color value to obtain the first color value of the third pixel.

14. An image processing apparatus, characterized in that, The image processing apparatus includes: The acquisition module is used to acquire data to be processed, wherein the multiple images in the data to be processed include images of the scene from multiple perspectives; The processing module is used to input the data to be processed into the first model, and perform feature extraction through the first model to obtain the feature information of each image in the data to be processed; The processing module is further configured to perform feature processing on each image in the data to be processed using the first model, and obtain the prediction result output by the first model. The first model is trained based on training samples and a first loss term. The training samples include multiple images of the first scene from multiple perspectives. The training samples include a first image and a second image. The first loss term indicates the similarity between first feature information and second feature information. The first feature information includes feature information of a first point in the first image. The second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. The same point is a three-dimensional spatial point. The first point is the projection of the three-dimensional spatial point onto the first image, and the second point is the projection of the three-dimensional spatial point onto the second image. The first model is trained based on training samples and a first loss term, including: training the first model according to the feature information of each image in the training samples, the first loss term, and the second loss term. The goal of training with the second loss term includes reducing the similarity between the first feature information and the third feature information. The third feature information includes the feature information of a third point in the second image. The third point is different from the second point. Both the second point and the third point are located on a first line. The second line is projected onto the second image to obtain the first line. The second line passes through the first point and passes through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image.

15. The apparatus according to claim 14, characterized in that, The feature information of each image in the data to be processed includes the feature information of multiple first pixels of each image in the data to be processed; The processing module is specifically used to obtain the initial feature information of the second pixel, and fuse the initial feature information of the first pixel with the initial feature information of the second pixel to obtain the feature information of the first pixel. The first pixel and the second pixel are points in different images in the data packet to be processed. The first pixel and the second pixel have the same semantics. The same semantics means that the first pixel and the second pixel are projections of the same three-dimensional space point on their respective images.

16. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire data to be processed, wherein the multiple images in the data to be processed include images of the scene from multiple perspectives; The processing module is used to input the data to be processed into the first model, and perform feature extraction through the first model to obtain the feature information of each image in the data to be processed; The processing module is further configured to perform feature processing on each image in the data to be processed using the first model, and obtain the prediction result output by the first model. The feature information of each image in the data to be processed includes the feature information of multiple first pixels of each image in the data to be processed. The processing module is specifically used to obtain the initial feature information of the second pixel, fuse the initial feature information of the first pixel with the initial feature information of the second pixel to obtain the feature information of the first pixel, wherein the first pixel and the second pixel are points in different images in the data packet to be processed, and the first pixel and the second pixel have the same semantics, wherein the same semantics means that the first pixel and the second pixel are projections of the same three-dimensional space point on their respective images. The first model is trained based on training samples, a first loss term, and a second loss term. The training samples include multiple images of a first scene from multiple viewpoints, including a first image and a second image. The first loss term indicates the similarity between first and second feature information. The first feature information includes feature information of a first point in the first image, and the second feature information includes feature information of a second point in the second image. The first point in the first image and the second point in the second image correspond to the same point in the first scene. The goal of training with the second loss term is to reduce the similarity between the first and third feature information. The third feature information includes feature information of a third point in the second image, which is different from the second point. Both the second point and the third point are located on a first line. The second line is projected onto the second image to obtain the first line. The second line passes through the first point and passes through the focal point of the camera that captured the first image and / or the origin of the camera coordinate system corresponding to the first image.

17. A computer program product, characterized in that, The computer program product includes a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 8.

19. An execution device, characterized in that, It includes a processor and a memory, wherein the processor is coupled to the memory. The memory is used to store programs; The processor is configured to execute a program in the memory, causing the execution device to perform the method as described in any one of claims 6 to 8.

20. A training device, characterized in that, It includes a processor and a memory, wherein the processor is coupled to the memory. The memory is used to store programs; The processor is configured to execute a program in the memory, causing the training device to perform the method as described in any one of claims 1 to 5.

Images