Visual data and image processing method and visual and image processing model training method

Abstract

The embodiment of the invention provides a visual data and image processing method and a visual and image processing model training method, and the visual data processing method comprises the steps: determining target visual data, and inputting the target visual data into a visual processing model; a depth map of the target visual data and an incident ray set are obtained through the visual processing model, incident rays in the incident ray set are the position point of a target object in the target visual data and the incident rays between pixel points of the target object, the position point is the point of the target object in the three-dimensional coordinate system, and the pixel points of the target object are the point of the target object in the three-dimensional coordinate system; the pixel points are points of the target object in the two-dimensional imaging plane; determining target camera parameters of the target camera model according to the incident light set; obtaining three-dimensional point cloud data corresponding to target visual data according to target camera parameters and the depth map; and camera internal reference estimation is used as assistance to improve the performance of depth estimation, so that the precision and reliability of depth estimation are improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to methods for processing visual data and images, and methods for training visual and image processing models. Background Art

[0002] Video devices are among the most widely used sensing devices for three-dimensional perception and understanding of urban scenes, and are widely used in urban governance. To establish a correlation between the two-dimensional image of a device location and the true three-dimensional scene, absolute depth information and camera parameter information are crucial. Compared to active sensor devices, image-based passive vision technologies offer advantages in terms of cost, size, operating range, and flexibility. Among these technologies, estimating scene depth from a single image offers advantages such as low cost, ease of deployment, high versatility, and wide applicability, making it a preferred solution for scene depth estimation.

[0003] However, depth maps alone cannot reconstruct a three-dimensional scene with absolute scale. They also rely on the camera's intrinsic parameter information. Estimating camera intrinsic parameters from a single image is usually performed independently of depth estimation, which leads to a lack of effective information interaction and collaborative optimization between the two, thereby limiting the accuracy and reliability of three-dimensional scene understanding. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a method for processing visual data. One or more embodiments of this specification also relate to a visual data processing device, a visual processing model training method and device, an image processing method, an image processing model training method, an image processing method applied to traffic scenes, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a method for processing visual data is provided, comprising:

[0006] determining target visual data, and inputting the target visual data into a visual processing model;

[0007] Using the visual processing model, obtaining a depth map and an incident light set of the target visual data, wherein the incident light in the incident light set is the incident light between a position point of the target object in the target visual data and a pixel point of the target object, the position point is a point of the target object in a three-dimensional coordinate system, and the pixel point is a point of the target object in a two-dimensional imaging plane;

[0008] determining target camera parameters of a target camera model based on the set of incident rays;

[0009] According to the target camera parameters and the depth map, three-dimensional point cloud data corresponding to the target visual data is obtained.

[0010] According to a second aspect of the embodiments of this specification, there is provided a visual data processing apparatus, comprising:

[0011] a data determination module, configured to determine target visual data and input the target visual data into a visual processing model;

[0012] a first obtaining module configured to obtain, by using the visual processing model, a depth map and an incident ray set of the target visual data, wherein the incident rays in the incident ray set are incident rays between a position point of a target object in the target visual data and a pixel point of the target object, the position point being a point of the target object in a three-dimensional coordinate system, and the pixel point being a point of the target object in a two-dimensional imaging plane;

[0013] a parameter determination module, configured to determine target camera parameters of a target camera model based on the incident light set;

[0014] The second acquisition module is configured to obtain three-dimensional point cloud data corresponding to the target visual data according to the target camera parameters and the depth map.

[0015] According to a third aspect of the embodiments of this specification, a visual processing model training method is provided, comprising:

[0016] Determining a visual data sample, a depth map label of the visual data sample, and an incident ray set label of the visual data sample, wherein the incident ray in the incident ray set label is an incident ray between a position point of a sample object in the visual data sample and a pixel point of the sample object, the position point being a point of the sample object in a three-dimensional coordinate system, and the pixel point being a point of the sample object in a two-dimensional imaging plane;

[0017] Inputting the visual data sample into a visual processing model to obtain a predicted depth map and a predicted incident light set of the visual data sample;

[0018] The visual processing model is trained based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

[0019] According to a fourth aspect of the embodiments of this specification, a visual processing model training device is provided, comprising:

[0020] a determination module configured to determine a visual data sample, a depth map label of the visual data sample, and an incident ray set label of the visual data sample, wherein the incident ray in the incident ray set label is an incident ray between a position point of a sample object in the visual data sample and a pixel point of the sample object, the position point being a point of the sample object in a three-dimensional coordinate system, and the pixel point being a point of the sample object in a two-dimensional imaging plane;

[0021] an acquisition module configured to input the visual data sample into a visual processing model to obtain a predicted depth map and a predicted incident ray set of the visual data sample;

[0022] A training module is configured to train the visual processing model based on the predicted depth map, the depth map label, the predicted incident light set, and the incident light set label.

[0023] According to a fifth aspect of the embodiments of this specification, there is provided an image processing method, including:

[0024] Determining a target image, and inputting the target image into an image processing model;

[0025] Using the image processing model, obtaining a depth map and an incident ray set of the target image, wherein the incident rays in the incident ray set are incident rays between a position point of a target object in the target image and a pixel point of the target object, the position point being a point of the target object in a three-dimensional coordinate system, and the pixel point being a point of the target object in a two-dimensional imaging plane;

[0026] determining target camera parameters of a target camera model based on the set of incident rays;

[0027] According to the target camera parameters and the depth map, three-dimensional point cloud data corresponding to the target image is obtained.

[0028] According to a sixth aspect of the embodiments of this specification, there is provided an image processing model training method, comprising:

[0029] Determining an image sample, a depth map label of the image sample, and an incident ray set label of the image sample, wherein the incident ray in the incident ray set label is the incident ray between a position point of the sample object in the image sample and a pixel point of the sample object, the position point is a point of the sample object in a three-dimensional coordinate system, and the pixel point is a point of the sample object in a two-dimensional imaging plane;

[0030] Inputting the image sample into an image processing model to obtain a predicted depth map and a predicted incident light set of the image sample;

[0031] The image processing model is trained based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

[0032] According to a seventh aspect of the embodiments of this specification, there is provided an image processing method applied to a traffic scene, comprising:

[0033] determining a target traffic image, and inputting the target traffic image into a traffic image processing model;

[0034] Using the traffic image processing model, obtaining a depth map and an incident ray set of the target traffic image, wherein incident rays in the incident ray set are incident rays between a position point of a target object in the target traffic image and a pixel point of the target object, the position point being a point of the target object in a three-dimensional coordinate system, and the pixel point being a point of the target object in a two-dimensional imaging plane;

[0035] Determining target camera parameters of a target camera model according to the incident light set, and obtaining three-dimensional point cloud data corresponding to the target traffic image according to the target camera parameters and the depth map;

[0036] Determine a traffic event corresponding to the three-dimensional point cloud data according to the three-dimensional point cloud data.

[0037] According to an eighth aspect of the embodiments of this specification, a computing device is provided, including:

[0038] memory and processor;

[0039] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned visual data processing method, visual processing model training method, image processing method, and image processing model training method are implemented.

[0040] According to the ninth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual data processing method, visual processing model training method, image processing method, and image processing model training method.

[0041] According to the tenth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual data processing method, visual processing model training method, image processing method, and image processing model training method.

[0042] One embodiment of the present specification provides a visual data processing method, including: determining target visual data, and inputting the target visual data into a visual processing model; using the visual processing model to obtain a depth map and an incident light set of the target visual data, wherein the incident light in the incident light set is the incident light between a position point of a target object in the target visual data and a pixel point of the target object, the position point is a point of the target object in a three-dimensional coordinate system, and the pixel point is a point of the target object in a two-dimensional imaging plane; based on the incident light set, determining target camera parameters of a target camera model; based on the target camera parameters and the depth map, obtaining three-dimensional point cloud data corresponding to the target visual data.

[0043] The visual data processing method provided in the embodiments of this specification integrates the depth estimation task and the camera intrinsic parameter estimation task into a unified framework by utilizing a visual processing model, and can simultaneously obtain the depth map and incident light set corresponding to a single image. The incident light set serves as an implicit representation of the camera parameters, and the corresponding camera intrinsic parameters can be obtained through the incident light set. That is, a bridge for joint optimization of depth estimation and camera intrinsic parameter estimation is constructed using the incident light set, and the performance of depth estimation is improved by using the camera intrinsic parameter estimation as an auxiliary, thereby improving the accuracy and reliability of depth estimation, making it more robust in various scenarios, and improving the accuracy of the depth map and the incident light set. On this basis, the three-dimensional point cloud data obtained using the depth map and incident light is also more accurate. When the three-dimensional point cloud data with absolute scale can be directly restored from a single image, the workload of the computer hardware can be reduced and the processing performance of the computer hardware can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram of a scenario of a visual data processing method provided by an embodiment of this specification;

[0045] Figure 2 is a flowchart of a visual data processing method provided by one embodiment of this specification;

[0046] Figure 3 This is a flowchart of a visual processing model training method provided by one embodiment of this specification;

[0047] Figure 4 This is a schematic diagram of a training process of a visual processing model training phase provided by an embodiment of this specification;

[0048] Figure 5 This is a schematic diagram of the reasoning process of a visual processing model in the reasoning stage provided by one embodiment of this specification;

[0049] Figure 6is a flowchart of an image processing method provided by one embodiment of this specification;

[0050] Figure 7 This is a flowchart of an image processing model training method provided by one embodiment of this specification;

[0051] Figure 8 This is a flowchart of an image processing method applied to a traffic scene provided by one embodiment of this specification;

[0052] Figure 9 This is a schematic diagram of the structure of a visual data processing device provided by one embodiment of this specification;

[0053] Figure 10 This is a structural diagram of a visual processing model training device provided by one embodiment of this specification;

[0054] Figure 11 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0055] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0056] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0058] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0059] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model. It is pre-trained on a large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as a large language model (LLM) and a multi-modal pre-training model.

[0060] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0061] First, the terms involved in one or more embodiments of this specification are explained.

[0062] Depth estimation is a technique in computer vision that aims to predict the distance or depth of a scene from an image or image sequence. Depth estimation can be achieved using a variety of methods, including binocular stereo matching, multi-view stereo vision, structured light, time-of-flight sensors, and data-driven methods based on deep learning.

[0063] Monocular depth estimation involves estimating the depth of a scene using a single image captured by a single camera. Because monocular images lack the disparity information provided by stereo vision, monocular depth estimation typically relies on image cues such as texture gradients, occlusions, perspective scaling, and atmospheric effects, as well as using machine learning or deep learning models to learn scene depth information from large amounts of data.

[0064] Absolute depth: Metric Depth. Absolute depth refers to the real-world distance of a point on an object's surface relative to a given reference plane (usually the camera's imaging plane) in three-dimensional space. Absolute depth is a quantitative measurement, typically expressed in units of length (such as meters or centimeters), that provides precise scale information for the three-dimensional structure of a scene.

[0065] Camera parameter estimation: To determine the relative transformation between the 3D geometric position of a point on an object's surface and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters include camera intrinsic parameters and camera extrinsic parameters. The process of calculating these parameters is called camera parameter estimation. Camera parameter estimation is crucial for tasks such as 3D reconstruction, camera calibration, image correction, and augmented reality.

[0066] Camera intrinsic parameters: Camera intrinsic parameters refer to the optical and geometric characteristics of the camera itself. These parameters include focal length, optical center (the location of the intersection with the optical axis on the image plane), and distortion coefficients (parameters used to describe and correct lens distortion). Intrinsic parameters are part of the camera imaging model, mapping points in the 3D world to the 2D image plane.

[0067] Incident ray set: This can be understood as the incident field. The incident field is the set of incident rays between a point in three-dimensional space and a pixel on the two-dimensional imaging plane. It is a pixel-level two-dimensional matrix, where each element is related to the coordinates of the pixel on the two-dimensional imaging plane, the camera's optical center, and the camera's focal length. The incident field can be considered an implicit representation of the camera's intrinsic parameters.

[0068] Random Sampling Consensus (RANSAC) is an iterative optimization algorithm that estimates the parameters of a mathematical model from a set of observed data that contains outliers. The algorithm aims to find optimal model parameters that fit likely inliers (data points that can be well explained by a set of model parameters) while ignoring outliers (data points that cannot be accommodated by the model).

[0069] Currently, the application of large-scale cloud computing and artificial intelligence technologies has provided the conditions for achieving city-level 3D scene understanding. Video equipment, a common sensing device in urban scenes, is widely used in urban governance to observe pedestrians, vehicles, and events within target scenes. Traditional equipment, with its two-dimensional images, lacks a connection to the actual three-dimensional scene, typically only enabling target recognition, detection, and tracking based on two-dimensional images. Projecting two-dimensional images into a real three-dimensional space is crucial for achieving 3D urban scene understanding. The core of this problem lies in the perception of absolute scene depth and the estimation of camera parameters, thereby establishing a bridge between the two-dimensional image coordinates and the three-dimensional coordinates in space.

[0070] Scene depth perception can be categorized as active or passive, depending on whether or not external energy is radiated. Compared to the highly accurate but expensive active ranging, image-based passive ranging technology only requires two-dimensional images under natural light, offering flexible implementation, low cost, and wide applicability. Passive ranging technologies primarily include monocular vision, binocular stereo vision, and multi-camera stereo vision. Monocular depth estimation, which requires a single camera and does not rely on specific camera configurations or geometric constraints, offers advantages such as low cost, ease of deployment, high versatility, and wide applicability, making it a preferred solution for scene depth estimation.

[0071] However, although the monocular depth estimation method based on deep learning can estimate the absolute depth of a three-dimensional scene, it is impossible to reconstruct a three-dimensional scene with absolute scale based solely on depth information itself, and it also relies on the camera's intrinsic parameter information. Compared with the traditional camera calibration method based on geometric calibration objects, the camera intrinsic parameter estimation method based on deep learning is more flexible and lightweight, and does not require clear calibration objects as a reference. However, it is usually performed independently of the monocular depth estimation, and the estimation mode of the camera intrinsic parameter is significantly different from the mode of depth estimation. This makes it difficult for information exchange and collaborative optimization between the two, thereby limiting the accuracy and reliability of three-dimensional reconstruction.

[0072] In response to the deficiencies in the above-mentioned methods, the embodiments of the present specification provide a method for training a visual processing model to realize a method for jointly estimating monocular depth and camera intrinsic parameters based on the incident field. The method introduces a collaborative optimization mechanism of the depth estimation module and the incident field estimation module through a neural network model with shared weights (i.e., a visual processing model), so that the visual processing model can predict the incident field corresponding to the image while predicting the absolute depth of a single image. In the model reasoning stage, the incident field is used as an implicit representation of the camera intrinsic parameters, and the camera intrinsic parameters are estimated by the RANSAC iterative optimization algorithm. The visual data processing method provided in the embodiments of this specification integrates monocular depth estimation and camera intrinsic parameter estimation into a unified framework, and through multi-task learning and collaborative optimization, it not only improves the accuracy and reliability of depth estimation, but also can reconstruct a three-dimensional point cloud with scale, which has important application value in the fields of three-dimensional reconstruction, virtual reality, augmented reality, etc.

[0073] In this specification, a visual data processing method is provided. This specification also involves a visual data processing device, a visual processing model training method and device, an image processing method and device, an image processing model training method and device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0074] See also Figure 1 , Figure 1 A schematic diagram of a scenario of a visual data processing method provided according to an embodiment of this specification is shown.

[0075] Specifically, the visual data processing method is implemented by using the terminal device 102 and the server 104. The terminal device 102 is used to send the target visual data to the server 104. For example, the target visual data is an image. Figure 1 In practical applications, the target visual data can also be video frames, so as to process the video and obtain the three-dimensional point cloud data corresponding to the video frames.

[0076] A visual data processing model is trained in the server 104, and the visual data processing model includes an encoding-decoding network; when the server 104 receives the target visual data sent by the end-side device 102, the target visual data is input into the visual data processing model to obtain a depth map and an incident field of the target visual data in the encoding-decoding network of the visual data processing model; the camera intrinsic parameters are obtained through the incident field, and then the three-dimensional point cloud data corresponding to the target visual data is obtained according to the depth map and the camera parameters. In actual applications, the functions provided by OpenCV, PCL (Point Cloud Library), Open3D, etc. can be used to directly process the depth map, and the three-dimensional point cloud data can be quickly generated in combination with the camera intrinsic parameters; thereby, the three-dimensional scene constructed based on the three-dimensional point cloud data is returned to the end-side device 102 for display on the end-side device 102.

[0077] The end-side device 102 may include a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5, version 5 of Hypertext Markup Language) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The end-side device may be based on the software development kit (SDK) of the corresponding service provided by the server, such as developed based on the real-time communication (RTC) SDK, etc. The end-side device can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. The electronic device may have a display screen and support information browsing, etc., such as a personal mobile terminal such as a mobile phone, tablet computer, personal computer, etc. Various other types of applications can usually be configured in the electronic device, such as human-computer dialogue applications, model training applications, visual data processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0078] Server 104 can be understood as a server that provides various services, including physical servers and cloud servers. For example, a server that provides communication services to multiple clients, a server that supports backend training for models used on clients, or a server that processes data sent by clients. It should be noted that server 104 can be implemented as a distributed server cluster consisting of multiple servers or as a single server. Server 104 can also be a server for a distributed system or a server integrated with blockchain. Server 104 can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0079] It is worth noting that the visual data processing method provided in the embodiments of this specification can be executed by the server 104. In other embodiments of this specification, the visual data processing model can be deployed in the terminal device 102, so that the terminal device 102 can also have similar functions as the server 104, thereby executing the visual data processing method provided in the embodiments of this specification; in other embodiments, the visual data processing method provided in the embodiments of this specification can also be jointly executed by the terminal device 102 and the server 104.

[0080] The visual data processing method provided in the embodiments of this specification integrates the depth estimation task and the camera intrinsic parameter estimation task into a unified framework by utilizing a visual processing model. It can simultaneously obtain the depth map and incident light set corresponding to a single image. The incident light set serves as an implicit representation of the camera parameters, and the corresponding camera intrinsic parameters can be obtained through the incident light set. That is, a bridge for joint optimization of depth estimation and camera intrinsic parameter estimation is constructed using the incident light set, and the performance of depth estimation is improved by using camera intrinsic parameter estimation as an auxiliary, thereby improving the accuracy and reliability of depth estimation, making it more robust in various scenarios, and improving the accuracy of the depth map and the incident light set. On this basis, the three-dimensional point cloud data obtained using the depth map and incident light is also more accurate, and three-dimensional point cloud data with absolute scale can be directly restored from a single image.

[0081] See also Figure 2 , Figure 2 A flowchart of a visual data processing method provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0082] Step 202: Determine target visual data, and input the target visual data into a visual processing model.

[0083] Among them, the target visual data can be understood as visual data such as a single image, a single video frame, etc. containing information of a specific object or scene. Of course, in practical applications, multiple images or multiple video frames can also be determined as target visual data; the visual processing model can be understood as a model for processing visual data, which can mine the information contained in the target visual data through the visual processing model, and obtain a depth map and a set of incident light corresponding to the target visual data.

[0084] Specifically, an image or video frame can be used as target visual data and input into a visual processing model to obtain a depth map and a set of incident light rays corresponding to the image or video frame using the visual processing model.

[0085] Step 204: Utilize the visual processing model to obtain a depth map and an incident light set of the target visual data, wherein the incident light in the incident light set is the incident light between the position point of the target object in the target visual data and the pixel point of the target object, the position point is the point of the target object in the three-dimensional coordinate system, and the pixel point is the point of the target object in the two-dimensional imaging plane.

[0086] Among them, the depth map can be understood as a type of image data that records the distance information from each pixel in the two-dimensional imaging plane to the center of the camera in the form of a grayscale image or color code. It is a way to express the three-dimensional spatial structure. The depth information about the three-dimensional scene can be obtained through the depth map.

[0087] The incident light set can be understood as the incident field, which refers to the set of incident light rays between a point in three-dimensional space and a pixel point in a two-dimensional imaging plane.

[0088] Specifically, a location point is a point in three-dimensional space. In a three-dimensional coordinate system, a location can be accurately represented by three coordinate values ​​(X, Y, Z). Each pixel in a two-dimensional imaging plane can typically be represented by two coordinate values ​​(u, v), where u and v correspond to the position of the horizontal and vertical axes, respectively. In some cases, it can also be described using (x, y) coordinates, where the x and y axes define the horizontal and vertical directions of the imaging plane.

[0089] When the three-dimensional space is projected onto a two-dimensional imaging plane, each point of the object in the three-dimensional space will be converted into pixel points on the two-dimensional projection. This conversion process can be achieved through various geometric projection methods, such as orthographic projection, perspective projection, etc.; during the conversion process, some depth information originally in the three-dimensional space may be lost, because the two-dimensional image cannot directly record the third-dimensional coordinates (Z-axis coordinates) of the points in the three-dimensional space, and each pixel value on the depth map represents the distance from the object at the pixel position in the corresponding three-dimensional scene to the camera.

[0090] In practical applications, in order to establish the association between a two-dimensional image and a real three-dimensional scene (a specific application or instance description based on three-dimensional space, which includes various elements in three-dimensional space, such as buildings, vegetation, lighting, texture and other information, which are combined into a three-dimensional picture that can be interacted or observed), absolute scale depth information and camera parameters are required. The camera intrinsic parameter matrix is ​​a matrix used to describe the optical and geometric characteristics of the camera in computer vision and photogrammetry; it maps points in the three-dimensional space under the camera coordinate system to two-dimensional points on the image plane (i.e., the two-dimensional imaging plane). The camera intrinsic parameter matrix is ​​obtained during the camera calibration process and is related to the properties of the camera itself. The following is prior knowledge:

[0091] Given a simple camera intrinsic parameter matrix K simple Definition: Definition of the standard camera intrinsic parameter matrix K: The inverse matrix K of the camera intrinsic parameters -1 Definition:

[0092] Among them, f x and f y Respectively represent the focal lengths in the x-axis and y-axis directions on the image plane (usually the sensor plane), in pixels. x and b y Represent the coordinates of the optical center of the x-axis and y-axis, that is, the point where the camera optical axis passes through the image plane. The [0 0 1] in the third row maintains the consistency of homogeneous coordinates, which is convenient for matrix operations. If the standard camera matrix K is simplified so that the focal lengths of the x-axis and y-axis are equal, and the optical center coordinates of the camera are set at the center of the image, the simple camera matrix K can be obtained. simple , where w and h represent the width and height of the two-dimensional image respectively. Since the standard camera intrinsic parameter matrix K is a 3-order square matrix, its inverse matrix K exists -1 , meet KK -1 =I, where I is the identity matrix.

[0093] Given a two-dimensional homogeneous coordinate point p T=[x, y, 1], and its depth value d=D(p), where p represents the coordinate of the pixel point in the two-dimensional image, D represents the depth map, and d represents the depth value of the depth map D at the pixel point p.

[0094] Projecting from three-dimensional space to two-dimensional space requires the camera intrinsic parameter matrix, which is in the following form: K·P=p, where P represents the three-dimensional coordinate point. Multiply both sides of this formula by the inverse matrix K of the camera intrinsic parameter. -1 , we can get Therefore, the three-dimensional coordinate point Right now

[0095] Among them, vector v is the incident light, which comes from the three-dimensional coordinate point, points to the two-dimensional image pixel point, and passes through the origin of the camera; therefore, the incident field is composed of the set of incident rays associated with each pixel, and the incident rays in the incident field can be represented by the camera intrinsic parameters; in this way, when the incident field is obtained later, the camera intrinsic parameters can be obtained through the incident field.

[0096] In one or more embodiments of this specification, the visual processing model includes a feature extraction unit, a depth map prediction unit, and an incident light set prediction unit. By utilizing the depth map prediction unit and the incident light set prediction unit, when an image is input to the visual processing model, the visual processing model can simultaneously predict the depth map and incident field of the image. The specific implementation method is as follows:

[0097] The obtaining of the depth map and incident light set of the target visual data by using the visual processing model includes:

[0098] Using the feature extraction unit, obtaining visual features of the target visual data;

[0099] Inputting the visual features into the depth map prediction unit and the incident light set prediction unit;

[0100] Obtaining a depth map of the target visual data using the depth map prediction unit;

[0101] The incident light set prediction unit is used to obtain the incident light set of the target visual data.

[0102] Among them, the visual processing model can be understood as a neural network model, which is constructed based on a monocular depth estimation network with an encoder-decoder network structure. The encoder-decoder network is not limited to a specific model structure and can be a convolutional neural network, such as ResNet, CovNext, etc., or a Transformer (a neural network model based on a self-attention mechanism) network, such as Swin Transformer, Visual Transformer, etc.; the monocular depth estimation network is not limited to a specific model structure and can be a monocular depth estimation network such as NewCRFs, iDisc, etc.; no limitation is made here.

[0103] Specifically, taking the target visual data as an image as an example, the feature extraction unit is the encoding network, which uses the encoding network to extract the image features of the input image, and the depth map prediction unit and the incident light set prediction unit are the decoding network; the decoding network is responsible for restoring the abstract high-level image features into a depth map output and an incident field output with the same size and resolution as the input image in the spatial dimension; if the extracted image features are input into the depth map prediction unit, the depth map prediction unit estimates and predicts the depth map corresponding to the image based on the image features, that is, the depth value of each pixel in the three-dimensional space; if the extracted image features are input into the incident light set prediction unit, the incident light set prediction unit estimates and predicts the incident field corresponding to the image based on the image features, which reflects the process of projecting three-dimensional points in a real three-dimensional scene onto a two-dimensional imaging plane.

[0104] Given an input image I and a monocular depth estimation network D based on an encoder-decoder structure θ (i.e., the feature extraction unit in the above embodiment), the depth map prediction unit f d and the incident field prediction branch f v Share the weight parameters of the monocular depth estimation network to predict the depth map d and the incident field v respectively: d = f d (D θ (I)), v = f v (D θ (I)).

[0105] In practical applications, the depth map prediction unit uses the image features output by the feature extraction unit as input, and uses the convolution layer to regress a single-channel pixel-level depth map, where a single channel means that the shape and size of the depth map is h*w*1, h and w are the height and width of the image respectively, and there is only one channel in the third dimension, so it is a single channel, and each pixel contains only one value, which represents the depth value; the incident light set prediction unit also uses the image features output by the feature extraction unit as input, and uses the convolution layer to regress a three-channel pixel-level incident field, and normalizes it according to the third channel at the same time, where three channels means that the shape and size of the incident field is h*w*1, and there are 3 channels in the third dimension, representing the incident field in the x-axis direction, the incident field in the y-axis direction, and the normalized dimension (the channel was originally the incident field in the z-axis direction, and the value of the z-axis was changed to 1, and the z-axis was normalized).

[0106] The visual data processing method provided in the embodiments of this specification is that during the forward propagation of the image in the visual processing model, both the depth map prediction unit and the incident light set prediction unit perform calculations on the same image feature representation and share the weight parameters of the monocular depth estimation network. In this way, the complexity of the visual processing model can be reduced, and the feature representation capabilities learned in solving the depth estimation task can be used to assist in the prediction of the incident field. This can improve the correlation and overall performance between the depth estimation task and the camera intrinsic parameter estimation task, thereby achieving more accurate three-dimensional visual understanding and reconstruction.

[0107] Step 206: determining target camera parameters of a target camera model according to the incident light set;

[0108] The camera model can be understood as a mathematical model used to describe the process of imaging from a three-dimensional scene to a two-dimensional image plane.

[0109] Specifically, according to the above prior knowledge, the incident field can be represented by camera parameters. Therefore, when the incident field is obtained, the camera parameters can be determined.

[0110] In one or more embodiments of this specification, when an incident light set is obtained, the target camera parameters can be obtained by a random sampling consensus algorithm, or by a target parameter processing model. The specific implementation method is as follows:

[0111] determining target camera parameters of a target camera model based on the incident light set;

[0112] Determining target camera parameters of a target camera model using a random sampling consensus algorithm based on the set of incident rays; or

[0113] The incident light set is input into a target parameter processing model corresponding to the target camera model to obtain target camera parameters of the target camera model.

[0114] Among them, the random sampling consensus algorithm is the RANSAC iterative optimization algorithm.

[0115] When a set of incident rays is obtained, multiple incident rays can be obtained, but there may be noise, outliers, or mismatched point pairs in these incident ray data. In this case, the RANSAC algorithm can be used. This algorithm continuously randomly extracts a small number of sample points and estimates the camera parameters based on these points. It then calculates the support of the remaining points for the estimated camera parameters (that is, how many points satisfy the estimated camera parameters). After multiple iterations, the estimated camera parameters that maximize the number of inliers are found, which are the optimal target camera parameters.

[0116] Alternatively, the incident light set is directly used as input and the target camera parameters are predicted through the trained target parameter processing model.

[0117] The visual data processing method provided in the embodiments of this specification can select a suitable method to obtain the target camera parameters according to the actual situation when obtaining the incident light set, which not only improves the accuracy of the target camera parameters, but also improves the overall performance and efficiency of the system.

[0118] In one or more embodiments of this specification, the target camera model includes a first target camera model, and the camera parameters of the first target camera model include a camera focal length. In this case, the target camera focal length of the first target camera model is obtained. The specific implementation is as follows:

[0119] Determining target camera parameters of a target camera model using a random sampling consensus algorithm based on the incident light set includes:

[0120] determining a preset camera focal length of the first target camera model;

[0121] determining a target camera focal length of the first target camera model according to the preset camera focal length and a target scoring function corresponding to the first target camera model;

[0122] Among them, the target scoring function is obtained by summing the first horizontal direction scoring function and the first vertical direction scoring function, the first horizontal direction scoring function is determined according to the current camera focal length, the width of the two-dimensional imaging plane, the set of two-dimensional coordinate points in the two-dimensional imaging plane, and the incident light set, and the first vertical direction scoring function is determined according to the current camera focal length, the height of the two-dimensional imaging plane, the set of two-dimensional coordinate points in the two-dimensional imaging plane, and the incident light set.

[0123] Among them, the first target camera model can be understood as the simple camera model in the above embodiment. Compared with the standard camera model, the camera focal lengths in the x and y directions in the simple camera model are equal, and the optical center is located at the center point of the image. Therefore, the camera parameter of the simple camera model is the camera focal length.

[0124] When using the RANSAC iterative optimization algorithm, a smaller solver is used to generate multiple candidate camera intrinsic parameter matrices, and the optimal solution that maximizes the scoring function is selected. The standard camera model requires solving the camera focal length and optical center in the x and y directions, while the simple camera model requires solving the camera focal length.

[0125] Candidate values ​​for the camera focal length are: Among them, f represents the predicted camera focal length, f min Represents the smaller focal length value among all candidate focal lengths, f max Represents the larger focal length value among all candidate focal lengths, N f Represents the number of candidate focal lengths, and i represents the subscript of each candidate focal length.

[0126] The scoring function under the simple camera model is defined as the sum of the x-direction and the y-direction: ρ(f, {x}, {v}) = ρ x (f x ,w / 2,{x},{v})+ρ y (f y , h / 2,{x},{v}), where ρ represents the scoring function, f represents the predicted camera focal length, {x} represents the set of two-dimensional coordinate points, {v} represents the set of incident field vectors, and ρ x represents the scoring function in the x direction, ρ y represents the scoring function in the y direction, w and h represent the width and height of the image respectively.

[0127] The visual data processing method provided in the embodiments of this specification can handle noise, outliers and inaccurate matches in the data by using the RANSAC iterative optimization algorithm. Through iterative random sampling and consistency detection, it can robustly estimate camera parameters in complex environments and avoid the impact of erroneous matching point pairs.

[0128] In one or more embodiments of this specification, when the RANSAC iterative optimization algorithm is an iterative algorithm, the current camera focal length is determined as the target camera focal length of the first target camera model when the function calculation result of the target scoring function satisfies the iteration stopping condition through iteration. The specific implementation method is as follows:

[0129] The determining, according to the preset camera focal length and a target scoring function corresponding to the first target camera model, a target camera focal length of the first target camera model includes:

[0130] Determine the preset camera focal length as the current camera focal length;

[0131] Calculating a function calculation result using a target scoring function corresponding to the first target camera model according to the current camera focal length;

[0132] When it is determined that the function calculation result satisfies the iteration stopping condition, determining the target camera focal length of the first target camera model according to the function calculation result and the current camera focal length;

[0133] When it is determined that the function calculation result does not meet the iteration stopping condition, the next current camera focal length is determined according to the current camera focal length, and the step of calculating the function calculation result according to the current camera focal length using the target scoring function corresponding to the first target camera model is continued.

[0134] The iteration stopping condition may be a preset number of iterations, such as stopping the iteration after 100 iterations; or it may be a situation where the results of multiple consecutive iterations have little change.

[0135] Specifically, first, the RANSAC process randomly selects a small number of data points (the data points are the incident light rays in the incident light set). These data points are considered to be "intrinsic" points generated from the ideal model, that is, they satisfy the camera model and are not affected by noise. The selected "intrinsic" points are used to calculate a candidate camera intrinsic parameter matrix, which includes the parameter of the camera focal length (that is, the preset camera focal length).

[0136] The preset camera focal length value is set as the current camera focal length. Based on the current camera focal length, the target scoring function corresponding to the first target camera model is used for calculation to obtain a function calculation result. The function calculation result reflects the performance of the camera model at the current camera focal length, such as calculating the number of "insider" points that can be correctly projected back within a certain threshold range.

[0137] Repeat the above steps, generating a new candidate camera intrinsic parameter matrix each time, and record the set of current camera focal lengths with the larger function calculation result of the target scoring function so far. When the preset number of iterations is reached or it is found that the function calculation result of the target scoring function has not been significantly improved for several consecutive times, the current camera focal length corresponding to the larger target scoring function is selected as the target camera focal length of the first target camera model.

[0138] The visual data processing method provided in the embodiments of this specification can effectively estimate the target camera intrinsic parameters in the presence of a large amount of noise through the RANSAC iterative optimization algorithm. Even if most data points may be "outlier" points caused by noise, it can still find those camera intrinsic parameters that can better explain most normal data, thereby improving overall robustness.

[0139] In one or more embodiments of this specification, the target camera model includes a second target camera model, and the camera parameters of the second target camera model include the horizontal camera focal length, the vertical camera focal length, the horizontal optical center coordinates, and the vertical optical center coordinates. In this case, the target camera intrinsic parameters obtained include four parameters. The specific implementation method is as follows:

[0140] Determining target camera parameters of a target camera model using a random sampling consensus algorithm based on the incident light set includes:

[0141] determining at least two preset incident rays of the second target camera model, wherein the at least two preset incident rays are determined from the incident ray set;

[0142] Determining initial camera parameters of the second target camera model according to the at least two preset incident light rays, wherein the initial camera parameters include a first initial camera focal length, a second initial camera focal length, a first initial optical center coordinate, and a second initial optical center coordinate;

[0143] Determine a first target camera focal length and a first target optical center coordinate according to the first initial camera focal length, the first initial optical center coordinate, and a second horizontal direction scoring function,

[0144] The second horizontal direction scoring function is determined according to the first current camera focal length, the first current optical center coordinates, the set of two-dimensional coordinate points in the two-dimensional imaging plane, and the set of incident light rays;

[0145] Determine the second target camera focal length and the second target optical center coordinates according to the second initial camera focal length, the second initial optical center coordinates, and the second vertical direction scoring function,

[0146] The second vertical direction scoring function is determined according to the second current camera focal length, the second current optical center coordinates, a set of two-dimensional coordinate points in the two-dimensional imaging plane, and the set of incident light rays.

[0147] Among them, the second target camera model is the standard camera model in the above embodiment; the first initial camera focal length can be understood as the initial camera focal length in the horizontal (x-axis) direction; the second initial camera focal length can be understood as the initial camera focal length in the vertical (y-axis) direction; the first initial optical center coordinate can be understood as the initial optical center coordinate in the horizontal (x-axis) direction; the second initial optical center coordinate can be understood as the initial optical center coordinate in the vertical (y-axis) direction.

[0148] Specifically, first according to the definition of the incident field, the relationship between the incident vector v and the camera intrinsic parameter K is as follows:

[0149]

[0150] Where v represents the incident field vector, T represents the transpose, and K -1 Represents the inverse matrix of the camera intrinsic parameter matrix K, x (letters in bold, K -1 The following x) represents a vector in two-dimensional space; x and y (letters are not bold) represent the index coordinates in the x direction and y direction of the two-dimensional coordinate space, respectively. x and b y Respectively represent the optical center coordinates in the x and y directions of the camera intrinsic parameters, f x and f y Respectively represent the camera focal length in the x and y directions in the camera intrinsic parameters.

[0151] The solver can be used to solve the camera internal parameters. In the predicted incident field, two incident vectors are randomly sampled. and Then the camera intrinsic parameters can be expressed as:

[0152]

[0153] Among them, b x and b y Respectively represent the optical center coordinates in the x and y directions of the camera intrinsic parameters, f x and f y Respectively represent the camera focal length in the x and y directions in the camera intrinsic parameters; 1 ,y 1 Represents one of the incident vectors v 1 The coordinate point in the two-dimensional space, x 2 ,y 2 Represents another incident vector v 2 The coordinate point in the two-dimensional space, and Represent the incident field vector v 1 The values ​​on the x-axis and y-axis, and Represent the incident field vector v 2The values ​​on the x-axis and y-axis.

[0154] The scoring functions in the x-direction and y-direction are defined as follows:

[0155] x-direction:

[0156] Among them, ρ x represents the scoring function in the x direction, b x Indicates the optical center coordinate in the x direction in the camera intrinsic parameter, f x Indicates the focal length of the camera in the x direction in the camera intrinsic parameter, {x} represents the set of two-dimensional coordinate points, {v} represents the set of incident field vectors, i represents the subscript of the point in the RANSAC algorithm, k x Indicates the threshold value of the x-direction as the internal point of the RANSAC algorithm, N k Represents the total number of inliers and outliers in the RANSAC algorithm, x represents the value of the two-dimensional coordinate point on the x-axis, v x Represents the value of the incident field vector on the x-axis.

[0157] y direction:

[0158] Among them, ρ y represents the scoring function in the y direction, b y Indicates the optical center coordinate in the y direction of the camera intrinsic parameter, f y Indicates the camera focal length in the y direction in the camera intrinsic parameter, k y Indicates the threshold value of the y-direction point judged as an inlier in the RANSAC algorithm, y represents the value of the two-dimensional coordinate point on the y-axis, v y Represents the value of the incident field vector on the y-axis.

[0159] The determining of the first target camera focal length and the first target optical center coordinates according to the first initial camera focal length, the first initial optical center coordinates, and the second horizontal direction scoring function includes:

[0160] Determine the first initial camera focal length as the first current camera focal length, and determine the first initial optical center coordinates as the first current optical center coordinates;

[0161] Obtaining a first function calculation result by calculating the second horizontal direction scoring function corresponding to the second target camera model according to the first current camera focal length and the first current optical center coordinates;

[0162] When it is determined that the calculation result of the first function satisfies the iteration stopping condition, determining the first target camera focal length and the first target optical center coordinates of the second target camera model according to the calculation result of the first function, the first current camera focal length, and the first current optical center coordinates;

[0163] When it is determined that the calculation result of the first function does not meet the iteration stopping condition, the next first current camera focal length and the next first current optical center coordinates are determined according to the first current camera focal length and the first current optical center coordinates, and the step of calculating the first function calculation result according to the first current camera focal length and the first current optical center coordinates using the second horizontal direction scoring function corresponding to the second target camera model is continued.

[0164] For details, please refer to the implementation of the first target camera model mentioned above. The difference between the second target camera model and the first target camera model is that the calculated camera intrinsic parameters are different. That is, the first target camera model needs to obtain the camera focal length, and the second target camera model needs to obtain the horizontal camera focal length, vertical camera focal length, horizontal optical center coordinates, and vertical optical center coordinates.

[0165] In one or more embodiments of this specification, the target camera parameters of the target camera model can also be obtained by inputting the incident light set into the target parameter processing model. The specific implementation method is as follows:

[0166] Inputting the incident light set into a target parameter processing model corresponding to the target camera model to obtain target camera parameters of the target camera model includes:

[0167] When the target camera model is a first target camera model, inputting the incident light set into a first parameter processing model corresponding to the first target camera model to determine target camera parameters of the first target camera model, wherein the camera parameters of the first target camera model include a camera focal length, and the first parameter processing model is trained using incident light set samples and camera focal length labels corresponding to the incident light set samples; or

[0168] In the case that the target camera model is a second target camera model, the incident light set is input into a second parameter processing model corresponding to the second target camera model, and the target camera parameters of the second target camera model are determined using the second parameter processing model, wherein the camera parameters of the second target camera model include the camera focal length in the horizontal direction, the camera focal length in the vertical direction, the optical center coordinates in the horizontal direction, and the optical center coordinates in the vertical direction, and the first parameter processing model is obtained through training through incident light set samples, the camera focal length labels in the horizontal direction corresponding to the incident light set samples, the camera focal length labels in the vertical direction corresponding to the incident light set samples, the optical center coordinate labels in the horizontal direction corresponding to the incident light set samples, and the optical center coordinate labels in the vertical direction corresponding to the incident light set samples.

[0169] Specifically, because the camera intrinsic parameters to be calculated for the first target camera model and the second target camera model are different, different target parameter processing models need to be used to obtain the camera intrinsic parameters corresponding to the target camera models.

[0170] In practical applications, since the first target camera model needs to obtain the camera focal length, the first parameter processing model corresponding to the first target camera model can be trained using the incident light set samples and the camera focal length labels corresponding to the incident light set samples. By inputting the obtained incident light set into the first parameter processing model, the target camera focal length corresponding to the first target camera model can be obtained.

[0171] When the camera intrinsic parameters obtained in the second target camera model include the horizontal camera focal length, the vertical camera focal length, the horizontal optical center coordinates, and the vertical optical center coordinates, the second parameter processing model corresponding to the second target camera model is trained using the incident light set samples, the horizontal camera focal length labels corresponding to the incident light set samples, the vertical camera focal length labels corresponding to the incident light set samples, the horizontal optical center coordinate labels corresponding to the incident light set samples, and the vertical optical center coordinate labels corresponding to the incident light set samples. Only in this way can the target camera parameters including the horizontal camera focal length, the vertical camera focal length, the horizontal optical center coordinates, and the vertical optical center coordinates be obtained when the incident light set is input into the second parameter processing model.

[0172] In practical applications, the first parameter processing model and the second parameter processing model can also be used as two different parameter processing units of the target parameter processing model, namely the first parameter processing unit and the second parameter processing unit, and a calling unit for calling the large language model is added to the target parameter processing model. In this case, when the incident light set is input into the target parameter processing model, prompt text corresponding to the incident light set is input, including but not limited to text related to the target camera model, such as "calculate camera parameters corresponding to the first target camera model" and "the calculated camera parameters must include the horizontal camera focal length and the vertical camera focal length". When the incident light set and the prompt text are input into the target parameter processing model, the large language model is called by the calling unit, and the prompt text is semantically analyzed by the large language model. Based on the analysis results (obtaining the target camera parameters corresponding to the first target camera model or obtaining the target camera parameters corresponding to the second target camera model), the incident light set is input into the corresponding parameter processing unit, so that the corresponding target camera parameters are obtained by using the first parameter processing unit or the second parameter processing unit.

[0173] The visual data processing method provided in the embodiments of this specification can correspond to different target parameter processing models for different target camera models, thereby obtaining the target camera parameters corresponding to the target camera model based on the incident light set using the target parameter processing model, thereby improving the accuracy of estimating the target camera parameters.

[0174] Step 208: Obtain three-dimensional point cloud data corresponding to the target visual data according to the target camera parameters and the depth map.

[0175] Among them, the three-dimensional point cloud can be understood as a rich and accurate data structure that describes the three-dimensional geometric shape of the real world. In three-dimensional space, three-dimensional point cloud data can be understood as a set of discrete points, each of which carries specific three-dimensional coordinate information (X, Y, Z). These coordinates represent the geometric features of the surface of an object or scene, and are a digital representation of the three-dimensional shape of the real world. In addition to the basic three-dimensional coordinates, each point can also carry additional information, such as color (RGB value), reflection intensity and other attributes.

[0176] In practical applications, various libraries and tools can be used to implement the process of obtaining 3D point cloud data using target camera parameters and depth maps, such as OpenCV, PCL (Point Cloud Library), Open3D, etc. These libraries provide ready-made functions that can directly process depth maps and quickly generate 3D point cloud data in combination with camera parameters, thereby constructing a 3D scene corresponding to the target visual data based on the obtained 3D point cloud data.

[0177] The visual data processing method provided in the embodiments of this specification can simultaneously predict a depth map containing absolute depth information and obtain camera parameters from a single image, thereby reconstructing a three-dimensional scene with absolute scale from a single image and realizing the conversion from two-dimensional plane information to three-dimensional point cloud information; and through the optimized algorithms and calling interfaces provided by the library, ready-made functions can be directly called to convert depth maps and camera parameters to three-dimensional point cloud data, avoiding the time and computing resource consumption of implementing the algorithm from the bottom layer, thereby greatly improving the efficiency of obtaining three-dimensional point cloud data.

[0178] The visual data processing method provided in the embodiments of this specification integrates the depth estimation task and the camera intrinsic parameter estimation task into a unified framework by utilizing a visual processing model, and can simultaneously obtain the depth map and incident light set corresponding to a single image. The incident light set serves as an implicit representation of the camera parameters, and the corresponding camera intrinsic parameters can be obtained through the incident light set. That is, a bridge for joint optimization of depth estimation and camera intrinsic parameter estimation is constructed using the incident light set, and the performance of depth estimation is improved by using the camera intrinsic parameter estimation as an auxiliary, thereby improving the accuracy and reliability of depth estimation, making it more robust in various scenarios, and improving the accuracy of the depth map and the incident light set. On this basis, the three-dimensional point cloud data obtained using the depth map and incident light is also more accurate. When the three-dimensional point cloud data with absolute scale can be directly restored from a single image, the workload of the computer hardware can be reduced and the processing performance of the computer hardware can be improved.

[0179] See also Figure 3 , Figure 3 A flowchart of a visual processing model training method provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0180] Step 302: Determine a visual data sample, a depth map label of the visual data sample, and an incident light set label of the visual data sample, wherein the incident light in the incident light set label is the incident light between the position point of the sample object in the visual data sample and the pixel point of the sample object, the position point is the point of the sample object in the three-dimensional coordinate system, and the pixel point is the point of the sample object in the two-dimensional imaging plane.

[0181] Step 304: Input the visual data sample into a visual processing model to obtain a predicted depth map and a predicted incident light set of the visual data sample.

[0182] The specific implementation can be found in the above embodiments, which will not be described again here.

[0183] Step 306: Train the visual processing model based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

[0184] In one or more embodiments, the visual processing model includes a depth map prediction unit and an incident ray set prediction unit; when the depth map prediction unit is used to obtain a predicted depth map, the depth map prediction unit is trained using the predicted depth map and the depth map label; when the incident ray set prediction unit is used to obtain a predicted incident ray set, the incident ray set prediction unit is trained using the predicted incident ray set and the incident ray set label; the specific implementation method is as follows:

[0185] The step of training the visual processing model according to the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label comprises:

[0186] Training a depth map prediction unit of the visual processing model according to the predicted depth map and the depth map label;

[0187] The incident light set prediction unit of the visual processing model is trained based on the predicted incident light set and the incident light set label.

[0188] For details, see Figure 4 , Figure 4 A schematic diagram of a training process of a visual processing model training phase provided by one embodiment of this specification is shown;

[0189] Step 402: Determine an image sample and input the image sample into an encoding-decoding network;

[0190] Step 404: Obtain a predicted depth map corresponding to the image sample using an encoding-decoding network;

[0191] Step 406: Obtain a predicted incident field corresponding to the image sample using the encoder-decoder network;

[0192] Step 408: Calculate a first loss function based on the predicted depth map and the true depth map, and use gradient backpropagation to calculate the gradient contribution of each parameter in the encoding-decoding network to the first loss function.

[0193] Step 410: Calculate a second loss function based on the predicted incident field and the actual incident field, and use gradient backpropagation to calculate the gradient contribution of each parameter in the encoding-decoding network to the second loss function.

[0194] Among them, the image sample can be understood as the visual data sample in the above embodiment; the real depth map can be understood as the depth map label in the above embodiment; the real incident field can be understood as the depth map label in the above embodiment; the encoding-decoding network can be understood as the visual processing model in the above embodiment; the gradient backpropagation is a core algorithm in the neural network training process, which is mainly used to calculate the gradient of each parameter in the model with respect to the loss function, and then update these parameters to optimize the model.

[0195] Specifically, a depth map prediction unit in the encoding-decoding network is used to obtain a predicted depth map corresponding to the image sample, and an incident light set prediction unit in the encoding-decoding network is used to obtain a predicted incident field corresponding to the image sample.

[0196] In practical applications, the monocular depth estimation branch (i.e., the depth map prediction unit) uses scale-invariant logarithmic loss to supervise training, and its loss function is defined as follows:

[0197]

[0198] Where d represents the predicted depth map, d* represents the depth map label, L represents the loss function (the first loss function), Δd represents the difference between the predicted depth map and the true depth map in the logarithmic domain; n represents the number of pixels in the depth map, λ represents the adjustment factor, i represents the pixel subscript in the depth map; log represents the logarithmic function.

[0199] The incident field branch (i.e., the incident ray set prediction unit) uses cosine similarity loss to supervise training, and its loss function is defined as follows:

[0200] The overall loss function is: L total =αL(d, d * )+βL(v,v * )

[0201] Where v represents the predicted incident field vector, v* represents the true incident field vector, L represents the loss function (the second loss function), T represents the vector transpose, n represents the number of incident field vectors, and i represents the subscript of each incident field vector; L total represents the overall training loss function for the joint estimation of the incident field and depth map, α represents the loss weight of the monocular depth estimation branch, L(d,d*) represents the loss function of the monocular depth estimation branch, β represents the loss weight of the incident field branch, and L(v,v*) represents the loss function of the incident field branch.

[0202] In this way, the depth map prediction unit is trained according to the first loss function of the predicted depth map and the depth map label, and the incident light set prediction unit is trained according to the second loss function of the predicted incident light set and the incident light set label, and then the visual processing model is obtained by training according to the first loss function and the second loss function.

[0203] The visual processing model training method provided in the embodiments of this specification, based on the absolute depth estimation method and the camera intrinsic parameter estimation method, uses the incident field as a bridge to achieve joint optimization learning and estimation of monocular depth and camera intrinsic parameters, so as to restore the absolute scale and shape of the three-dimensional scene from a single image; the depth estimation task and the camera intrinsic parameter estimation task are integrated into a unified framework, providing a flexible and convenient solution for perceiving the three-dimensional scene structure; and the two tasks share network weights to promote each other, thereby improving the accuracy and reliability of monocular depth estimation, making it more robust in various scenarios.

[0204] See also Figure 5 , Figure 5 A schematic diagram of the reasoning process of a visual processing model in the reasoning stage provided by one embodiment of this specification is shown;

[0205] Step 502: Determine a target image and input the target image into an encoding-decoding network;

[0206] Step 504: Obtain a depth map of the target image using an encoding-decoding network;

[0207] Step 506: Obtain the incident field of the target image using the encoder-decoder network;

[0208] Step 508: Determine the camera intrinsic parameters of the target camera model using the RANSAC iterative optimization algorithm according to the incident field;

[0209] Step 510: Obtain a three-dimensional point cloud corresponding to the target image based on the camera intrinsic parameters and the depth map.

[0210] Specifically, a detailed description is given by taking the estimation of camera intrinsic parameters from the incident field by using the RANSAC iterative optimization algorithm as an example.

[0211] In practical applications, when an image is input into the encoding-decoding network, the depth map containing depth information and the incident field corresponding to the image can be obtained at the same time. Based on the incident field, the camera intrinsic parameters corresponding to the target camera model can be determined using the RANSAC iterative optimization algorithm. When depth information and camera intrinsic parameters are necessary conditions for constructing a three-dimensional scene, the three-dimensional point cloud corresponding to the image is obtained based on the depth map and camera intrinsic parameters.

[0212] The visual processing model provided in the embodiments of this specification integrates monocular depth estimation and camera intrinsic parameter estimation into a unified framework. Through multi-task learning and collaborative optimization, it not only improves the accuracy and reliability of depth estimation, but also can reconstruct a three-dimensional point cloud with scale. This has important application value in the fields of three-dimensional reconstruction, virtual reality, augmented reality, etc.

[0213] See also Figure 6 , Figure 6 A flowchart of an image processing method provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0214] Step 602: Determine a target image and input the target image into an image processing model;

[0215] Step 604: Using the image processing model, obtain a depth map and an incident ray set of the target image, wherein the incident rays in the incident ray set are incident rays between a position point of the target object in the target image and a pixel point of the target object, the position point being a point of the target object in a three-dimensional coordinate system, and the pixel point being a point of the target object in a two-dimensional imaging plane;

[0216] Step 606: Determine target camera parameters of the target camera model according to the incident light set;

[0217] Step 608: Obtain three-dimensional point cloud data corresponding to the target image according to the target camera parameters and the depth map.

[0218] A specific description is given by taking the target visual data as image data as an example. For specific implementation, please refer to the above embodiment, which will not be repeated here.

[0219] The image data processing method provided in the embodiments of this specification integrates the depth estimation task and the camera intrinsic parameter estimation task into a unified framework by utilizing an image processing model. It can simultaneously obtain the depth map and incident ray set corresponding to a single image. The incident ray set serves as an implicit representation of the camera parameters, and the corresponding camera intrinsic parameters can be obtained through the incident ray set. That is, a bridge for joint optimization of depth estimation and camera intrinsic parameter estimation is constructed using the incident ray set. The camera intrinsic parameter estimation is used as an auxiliary to improve the performance of depth estimation, thereby improving the accuracy and reliability of depth estimation, making it more robust in various scenarios, and improving the accuracy of the depth map and the incident ray set. On this basis, the three-dimensional point cloud data obtained using the depth map and incident light is also more accurate. When the three-dimensional point cloud data with absolute scale can be directly restored from a single image, the workload of computer hardware can be reduced and the processing performance of computer hardware can be improved.

[0220] See also Figure 7 , Figure 7 A flowchart of an image processing model training method provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0221] Step 702: Determine an image sample, a depth map label of the image sample, and an incident ray set label of the image sample, wherein the incident ray in the incident ray set label is the incident ray between a position point of the sample object in the image sample and a pixel point of the sample object, the position point is a point of the sample object in a three-dimensional coordinate system, and the pixel point is a point of the sample object in a two-dimensional imaging plane;

[0222] Step 704: Input the image sample into an image processing model to obtain a predicted depth map and a predicted incident light set of the image sample;

[0223] Step 706: Train the image processing model based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

[0224] The specific implementation can be found in the above embodiments, which will not be described again here.

[0225] The image processing model training method provided in the embodiments of this specification, based on the absolute depth estimation method and the camera intrinsic parameter estimation method, uses the incident field as a bridge to achieve joint optimization learning and estimation of monocular depth and camera intrinsic parameters, so as to achieve the ability to recover the absolute scale and shape of the three-dimensional scene from a single image; the depth estimation task and the camera intrinsic parameter estimation task are integrated into a unified framework, providing a flexible and convenient solution for perceiving the three-dimensional scene structure; and the two tasks share network weights to promote each other, thereby improving the accuracy and reliability of monocular depth estimation, so that more accurate three-dimensional point cloud data can be obtained using the image processing model.

[0226] See also Figure 8 , Figure 8 A flowchart of an image processing method applied to a traffic scene provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0227] Step 802: Determine a target traffic image, and input the target traffic image into a traffic image processing model.

[0228] Step 804: Utilize the traffic image processing model to obtain a depth map and an incident light set of the target traffic image, wherein the incident light in the incident light set is the incident light between the position point of the target object in the target traffic image and the pixel point of the target object, the position point is the point of the target object in the three-dimensional coordinate system, and the pixel point is the point of the target object in the two-dimensional imaging plane.

[0229] In one or more embodiments of the present specification, the traffic image processing model includes a feature extraction unit, a depth map prediction unit, and an incident ray set prediction unit; and obtaining the depth map and incident ray set of the target traffic image using the traffic image processing model includes:

[0230] Using the feature extraction unit, obtaining image features of the target traffic image;

[0231] Inputting the image features into the depth map prediction unit and the incident light set prediction unit;

[0232] Obtaining a depth map of the target traffic image using the depth map prediction unit;

[0233] The incident light set prediction unit is used to obtain the incident light set of the target traffic image.

[0234] For the specific implementation, please refer to the above embodiment and will not be described in detail here.

[0235] Step 806: Determine target camera parameters of a target camera model based on the incident light set, and obtain three-dimensional point cloud data corresponding to the target traffic image based on the target camera parameters and the depth map.

[0236] Step 808: Determine a traffic event corresponding to the three-dimensional point cloud data based on the three-dimensional point cloud data.

[0237] Specifically, when the image processing method is applied to a traffic scene, a specific traffic event can be determined based on the obtained three-dimensional point cloud data.

[0238] For example, in the field of autonomous driving, processing 3D point cloud data can detect various elements in the road environment, including but not limited to vehicles, pedestrians, bicycles, traffic signs, road markings, and obstacles. First, different types of objects, such as cars and pedestrians, can be identified from the 3D point cloud data. Detected targets are then tracked through a series of video frames to construct motion trajectories, analyzing information such as their direction and speed. Based on the detected object states, relationships, and traffic regulations, potential traffic events, such as congestion ahead, accidents, illegal parking, and emergency braking, can be determined.

[0239] Alternatively, in urban traffic intersection scenarios, by analyzing three-dimensional point cloud data, the positions, sizes, and relative positions of different vehicles can be accurately identified; this helps to achieve real-time vehicle flow statistics at intersections, abnormal congestion analysis, and potential accident warnings; further, the point cloud changes of three-dimensional point cloud data at consecutive time points can be used to detect whether the proximity between vehicles exceeds the safe distance, thereby predicting possible traffic accidents and issuing timely warnings.

[0240] Of course, in practical applications, the image processing method provided in the embodiments of this specification can be applied to traffic scenes, and can also be applied to various scenes with dome cameras or gun cameras, such as highway scenes, urban monitoring scenes, etc.

[0241] For example, in an urban surveillance scenario, an image or video frame captured by a gun camera is input into a pre-trained surveillance image processing model (i.e., the image processing model in the above embodiment), where the image or video frame captures a scene of a public place such as a park, commercial street or square, which contains target objects such as vehicles, pedestrians, road signs, and public facilities.

[0242] Using the monitoring image processing model, the depth information of each target object in three-dimensional space can be analyzed and calculated from the image or video frame, that is, a depth map can be generated, and the set of incident light rays between each position point (such as the center point of a vehicle, the position point of a pedestrian's head) and its pixel point on the image or video frame can be determined.

[0243] Based on these incident light sets, the parameters of the target camera used to capture the image or video frame can be reversely calculated, including the focal length, optical center coordinates, etc.; then, combined with the depth map, three-dimensional point cloud data corresponding to the image or video frame can be further generated. Each three-dimensional point cloud data represents the position of a pixel point in the image or video frame in the real three-dimensional space.

[0244] The resulting three-dimensional point cloud data is further analyzed and identified to identify various events, such as crowd density analysis (whether a gathering has occurred), behavior analysis (whether there is suspicious activity or inappropriate behavior), and environmental monitoring (whether there are any items left behind or facilities damaged). This information helps improve urban management efficiency, ensure public safety, and enable rapid responses to emergencies.

[0245] The image processing method applied to traffic scenes provided in the embodiments of this specification can improve urban traffic management efficiency and traffic safety in traffic scenes by understanding and processing three-dimensional point cloud data.

[0246] Corresponding to the above method embodiment, this specification also provides a visual data processing device embodiment, Figure 9 FIG. 1 shows a schematic diagram of the structure of a visual data processing device provided by an embodiment of this specification. Figure 9 As shown, the device includes:

[0247] a data determination module 902 configured to determine target visual data and input the target visual data into a visual processing model;

[0248] A first obtaining module 904 is configured to obtain a depth map and an incident ray set of the target visual data using the visual processing model, wherein the incident rays in the incident ray set are incident rays between a position point of the target object in the target visual data and a pixel point of the target object, the position point being a point of the target object in a three-dimensional coordinate system, and the pixel point being a point of the target object in a two-dimensional imaging plane;

[0249] a parameter determination module 906 configured to determine target camera parameters of a target camera model based on the incident light set;

[0250] The second obtaining module 908 is configured to obtain three-dimensional point cloud data corresponding to the target visual data according to the target camera parameters and the depth map.

[0251] The device further comprises:

[0252] a training module configured to determine a visual data sample, a depth map label of the visual data sample, and an incident ray set label of the visual data sample, wherein the incident ray in the incident ray set label is an incident ray between a position point of an object sample in the visual data sample and a pixel point of the object sample, the position point being a point of the object sample in a three-dimensional coordinate system, and the pixel point being a point of the object sample in a two-dimensional imaging plane;

[0253] Inputting the visual data sample into a visual processing model to obtain a predicted depth map and a predicted incident light set of the visual data sample;

[0254] The visual processing model is trained based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

[0255] Optionally, the first obtaining module 904 is further configured to:

[0256] Using the feature extraction unit, obtaining visual features of the target visual data;

[0257] Inputting the visual features into the depth map prediction unit and the incident light set prediction unit;

[0258] Obtaining a depth map of the target visual data using the depth map prediction unit;

[0259] The incident light set prediction unit is used to obtain the incident light set of the target visual data.

[0260] Optionally, the parameter determination module 906 is further configured to:

[0261] Determining target camera parameters of a target camera model using a random sampling consensus algorithm based on the set of incident rays; or

[0262] The incident light set is input into a target parameter processing model corresponding to the target camera model to obtain target camera parameters of the target camera model.

[0263] Optionally, the parameter determination module 906 is further configured to:

[0264] determining a preset camera focal length of the first target camera model;

[0265] A target camera focal length of the first target camera model is determined according to the preset camera focal length and a target scoring function corresponding to the first target camera model.

[0266] Optionally, the parameter determination module 906 is further configured to:

[0267] Determine the preset camera focal length as the current camera focal length;

[0268] Calculating a function calculation result using a target scoring function corresponding to the first target camera model according to the current camera focal length;

[0269] When it is determined that the function calculation result satisfies the iteration stopping condition, determining the target camera focal length of the first target camera model according to the function calculation result and the current camera focal length;

[0270] When it is determined that the function calculation result does not meet the iteration stopping condition, the next current camera focal length is determined according to the current camera focal length, and the step of calculating the function calculation result according to the current camera focal length using the target scoring function corresponding to the first target camera model is continued.

[0271] Optionally, the parameter determination module 906 is further configured to:

[0272] determining at least two preset incident rays of the second target camera model, wherein the at least two preset incident rays are determined from the incident ray set;

[0273] Determining initial camera parameters of the second target camera model according to the at least two preset incident light rays, wherein the initial camera parameters include a first initial camera focal length, a second initial camera focal length, a first initial optical center coordinate, and a second initial optical center coordinate;

[0274] Determine a first target camera focal length and a first target optical center coordinate according to the first initial camera focal length, the first initial optical center coordinate, and a second horizontal direction scoring function,

[0275] The second horizontal direction scoring function is determined according to the first current camera focal length, the first current optical center coordinates, the set of two-dimensional coordinate points in the two-dimensional imaging plane, and the set of incident light rays;

[0276] Determine the second target camera focal length and the second target optical center coordinates according to the second initial camera focal length, the second initial optical center coordinates, and the second vertical direction scoring function,

[0277] The second vertical direction scoring function is determined according to the second current camera focal length, the second current optical center coordinates, a set of two-dimensional coordinate points in the two-dimensional imaging plane, and the set of incident light rays.

[0278] Optionally, the parameter determination module 906 is further configured to:

[0279] Determine the first initial camera focal length as the first current camera focal length, and determine the first initial optical center coordinates as the first current optical center coordinates;

[0280] Obtaining a first function calculation result by calculating the second horizontal direction scoring function corresponding to the second target camera model according to the first current camera focal length and the first current optical center coordinates;

[0281] When it is determined that the calculation result of the first function satisfies the iteration stopping condition, determining the first target camera focal length and the first target optical center coordinates of the second target camera model according to the calculation result of the first function, the first current camera focal length, and the first current optical center coordinates;

[0282] When it is determined that the calculation result of the first function does not meet the iteration stopping condition, the next first current camera focal length and the next first current optical center coordinates are determined according to the first current camera focal length and the first current optical center coordinates, and the step of calculating the first function calculation result according to the first current camera focal length and the first current optical center coordinates using the second horizontal direction scoring function corresponding to the second target camera model is continued.

[0283] Optionally, the parameter determination module 906 is further configured to:

[0284] Determine the second initial camera focal length as the second current camera focal length, and determine the second initial optical center coordinates as the second current optical center coordinates;

[0285] Obtaining a second function calculation result by calculating the second vertical direction scoring function corresponding to the second target camera model according to the second current camera focal length and the second current optical center coordinates;

[0286] When it is determined that the calculation result of the second function satisfies the iteration stopping condition, determining the second target camera focal length and the second target optical center coordinates of the second target camera model according to the calculation result of the second function, the second current camera focal length, and the second current optical center coordinates;

[0287] When it is determined that the calculation result of the second function does not meet the iteration stopping condition, the next second current camera focal length and the next second current optical center coordinates are determined according to the second current camera focal length and the second current optical center coordinates, and the step of calculating the second function calculation result according to the second current camera focal length and the second current optical center coordinates using the second vertical direction scoring function corresponding to the second target camera model is continued.

[0288] Optionally, the second obtaining module 908 is further configured to:

[0289] When the target camera model is a first target camera model, inputting the incident light set into a first parameter processing model corresponding to the first target camera model to determine target camera parameters of the first target camera model, wherein the camera parameters of the first target camera model include a camera focal length, and the first parameter processing model is trained using incident light set samples and camera focal length labels corresponding to the incident light set samples; or

[0290] In the case that the target camera model is a second target camera model, the incident light set is input into a second parameter processing model corresponding to the second target camera model, and the target camera parameters of the second target camera model are determined using the second parameter processing model, wherein the camera parameters of the second target camera model include the camera focal length in the horizontal direction, the camera focal length in the vertical direction, the optical center coordinates in the horizontal direction, and the optical center coordinates in the vertical direction, and the first parameter processing model is obtained through training through incident light set samples, the camera focal length labels in the horizontal direction corresponding to the incident light set samples, the camera focal length labels in the vertical direction corresponding to the incident light set samples, the optical center coordinate labels in the horizontal direction corresponding to the incident light set samples, and the optical center coordinate labels in the vertical direction corresponding to the incident light set samples.

[0291] The above is a schematic diagram of a visual data processing device according to this embodiment. It should be noted that the technical solution of this visual data processing device and the technical solution of the aforementioned visual data processing method are based on the same concept. For details not described in detail in the technical solution of the visual data processing device, please refer to the description of the technical solution of the aforementioned visual data processing method.

[0292] Corresponding to the above method embodiment, this specification also provides an embodiment of a visual processing model training device, Figure 10 FIG. 1 shows a schematic diagram of a visual processing model training device provided by an embodiment of this specification. Figure 10 As shown, the device includes:

[0293] a determination module 1002 configured to determine a visual data sample, a depth map label of the visual data sample, and an incident ray set label of the visual data sample, wherein the incident ray in the incident ray set label is an incident ray between a position point of a sample object in the visual data sample and a pixel point of the sample object, the position point being a point of the sample object in a three-dimensional coordinate system, and the pixel point being a point of the sample object in a two-dimensional imaging plane;

[0294] an obtaining module 1004 configured to input the visual data sample into a visual processing model to obtain a predicted depth map and a predicted incident ray set of the visual data sample;

[0295] The training module 1006 is configured to train the visual processing model based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

[0296] Optionally, the training module 1006 is further configured to:

[0297] Training a depth map prediction unit of the visual processing model according to the predicted depth map and the depth map label;

[0298] The incident light set prediction unit of the visual processing model is trained based on the predicted incident light set and the incident light set label.

[0299] The above is a schematic diagram of a visual processing model training device according to this embodiment. It should be noted that the technical solution of the visual processing model training device and the technical solution of the visual processing model training method described above are based on the same concept. For details not described in detail in the technical solution of the visual processing model training device, please refer to the description of the technical solution of the visual processing model training method described above.

[0300] Figure 11 The following is a block diagram of a computing device 1100 according to one embodiment of the present disclosure. Components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.

[0301] The computing device 1100 also includes an access device 1140 that enables the computing device 1100 to communicate via one or more networks 1160. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0302] In one embodiment of the present specification, the above components of the computing device 1100 and Figure 11 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 11 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0303] Computing device 1100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1100 may also be a mobile or stationary server.

[0304] Among them, the processor 1120 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned visual data processing method, visual processing model training method, image processing method, and image processing model training method.

[0305] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the computing device embodiment is generally similar to the visual data processing method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the visual data processing method embodiment.

[0306] One embodiment of this specification also provides a computer-readable storage medium, which stores a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual data processing method, visual processing model training method, image processing method, and image processing model training method.

[0307] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from the other embodiments. In particular, the computer-readable storage medium embodiment is generally similar to the visual data processing method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the visual data processing method embodiment.

[0308] One embodiment of this specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual data processing method, visual processing model training method, image processing method, and image processing model training method.

[0309] The above is a schematic diagram of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the aforementioned visual data processing method are based on the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the aforementioned visual data processing method.

[0310] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0311] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0312] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0313] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0314] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for processing visual data, comprising: determining target visual data, and inputting the target visual data into a visual processing model; Using the visual processing model, obtaining a depth map and an incident light set of the target visual data, wherein the incident light in the incident light set is the incident light between a position point of the target object in the target visual data and a pixel point of the target object, the position point is a point of the target object in a three-dimensional coordinate system, and the pixel point is a point of the target object in a two-dimensional imaging plane; determining target camera parameters of a target camera model based on the set of incident rays; According to the target camera parameters and the depth map, three-dimensional point cloud data corresponding to the target visual data is obtained.

2. The visual data processing method according to claim 1, wherein the visual processing model is obtained by training through the following steps: Determining visual data samples, depth map labels for the visual data samples, and incident ray set labels for the visual data samples, where The incident light in the incident light set label is the incident light between the position point of the object sample in the visual data sample and the pixel point of the object sample, the position point is the point of the object sample in the three-dimensional coordinate system, and the pixel point is the point of the object sample in the two-dimensional imaging plane; Inputting the visual data sample into a visual processing model to obtain a predicted depth map and a predicted incident light set of the visual data sample; The visual processing model is trained based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

3. The visual data processing method according to claim 1, wherein the visual processing model comprises a feature extraction unit, a depth map prediction unit, and an incident light set prediction unit; The obtaining of the depth map and incident light set of the target visual data by using the visual processing model includes: Using the feature extraction unit, obtaining visual features of the target visual data; Inputting the visual features into the depth map prediction unit and the incident light set prediction unit; Obtaining a depth map of the target visual data using the depth map prediction unit; The incident light set prediction unit is used to obtain the incident light set of the target visual data.

4. The visual data processing method according to claim 1, wherein the target camera parameters of the target camera model are determined based on the incident light set; Determining target camera parameters of a target camera model using a random sampling consensus algorithm based on the set of incident rays; or The incident light set is input into a target parameter processing model corresponding to the target camera model to obtain target camera parameters of the target camera model.

5. The visual data processing method according to claim 4, wherein the target camera model comprises a first target camera model, and the camera parameters of the first target camera model include a camera focal length; Determining target camera parameters of a target camera model using a random sampling consensus algorithm based on the incident light set includes: determining a preset camera focal length of the first target camera model; A target camera focal length of the first target camera model is determined according to the preset camera focal length and a target scoring function corresponding to the first target camera model.

6. The visual data processing method according to claim 5, wherein determining the target camera focal length of the first target camera model based on the preset camera focal length and a target scoring function corresponding to the first target camera model comprises: Determine the preset camera focal length as the current camera focal length; Calculating a function calculation result using a target scoring function corresponding to the first target camera model according to the current camera focal length; When it is determined that the function calculation result satisfies the iteration stopping condition, determining the target camera focal length of the first target camera model according to the function calculation result and the current camera focal length; When it is determined that the function calculation result does not meet the iteration stopping condition, the next current camera focal length is determined according to the current camera focal length, and the step of calculating the function calculation result according to the current camera focal length using the target scoring function corresponding to the first target camera model is continued.

7. The visual data processing method according to claim 4, wherein the target camera model comprises a second target camera model, and the camera parameters of the second target camera model comprise a horizontal camera focal length, a vertical camera focal length, a horizontal optical center coordinate, and a vertical optical center coordinate; Determining target camera parameters of a target camera model using a random sampling consensus algorithm based on the incident light set includes: determining at least two preset incident rays of the second target camera model, wherein the at least two preset incident rays are determined from the incident ray set; Determining initial camera parameters of the second target camera model according to the at least two preset incident light rays, wherein the initial camera parameters include a first initial camera focal length, a second initial camera focal length, a first initial optical center coordinate, and a second initial optical center coordinate; Determine a first target camera focal length and a first target optical center coordinate according to the first initial camera focal length, the first initial optical center coordinate, and a second horizontal direction scoring function, The second horizontal direction scoring function is determined according to the first current camera focal length, the first current optical center coordinates, the set of two-dimensional coordinate points in the two-dimensional imaging plane, and the set of incident light rays; Determine the second target camera focal length and the second target optical center coordinates according to the second initial camera focal length, the second initial optical center coordinates, and the second vertical direction scoring function, The second vertical direction scoring function is determined according to the second current camera focal length, the second current optical center coordinates, a set of two-dimensional coordinate points in the two-dimensional imaging plane, and the set of incident light rays.

8. The visual data processing method according to claim 7, wherein determining the first target camera focal length and the first target optical center coordinates according to the first initial camera focal length, the first initial optical center coordinates, and the second horizontal direction scoring function comprises: Determine the first initial camera focal length as the first current camera focal length, and determine the first initial optical center coordinates as the first current optical center coordinates; Obtaining a first function calculation result by calculating the second horizontal direction scoring function corresponding to the second target camera model according to the first current camera focal length and the first current optical center coordinates; When it is determined that the calculation result of the first function satisfies the iteration stopping condition, determining the first target camera focal length and the first target optical center coordinates of the second target camera model according to the calculation result of the first function, the first current camera focal length, and the first current optical center coordinates; When it is determined that the calculation result of the first function does not meet the iteration stopping condition, the next first current camera focal length and the next first current optical center coordinates are determined according to the first current camera focal length and the first current optical center coordinates, and the step of calculating the first function calculation result according to the first current camera focal length and the first current optical center coordinates using the second horizontal direction scoring function corresponding to the second target camera model is continued.

9. The visual data processing method according to claim 7, wherein determining the second target camera focal length and the second target optical center coordinates according to the second initial camera focal length, the second initial optical center coordinates, and the second vertical direction scoring function comprises: Determine the second initial camera focal length as the second current camera focal length, and determine the second initial optical center coordinates as the second current optical center coordinates; Obtaining a second function calculation result by calculating the second vertical direction scoring function corresponding to the second target camera model according to the second current camera focal length and the second current optical center coordinates; When it is determined that the calculation result of the second function satisfies the iteration stopping condition, determining the second target camera focal length and the second target optical center coordinates of the second target camera model according to the calculation result of the second function, the second current camera focal length, and the second current optical center coordinates; When it is determined that the calculation result of the second function does not meet the iteration stopping condition, the next second current camera focal length and the next second current optical center coordinates are determined according to the second current camera focal length and the second current optical center coordinates, and the step of calculating the second function calculation result according to the second current camera focal length and the second current optical center coordinates using the second vertical direction scoring function corresponding to the second target camera model is continued.

10. The visual data processing method according to claim 4, wherein the step of inputting the incident light set into a target parameter processing model corresponding to the target camera model to obtain target camera parameters of the target camera model comprises: When the target camera model is a first target camera model, inputting the incident light set into a first parameter processing model corresponding to the first target camera model to determine target camera parameters of the first target camera model, wherein the camera parameters of the first target camera model include a camera focal length, and the first parameter processing model is trained using incident light set samples and camera focal length labels corresponding to the incident light set samples; or In the case that the target camera model is a second target camera model, the incident light set is input into a second parameter processing model corresponding to the second target camera model, and the target camera parameters of the second target camera model are determined using the second parameter processing model, wherein the camera parameters of the second target camera model include the camera focal length in the horizontal direction, the camera focal length in the vertical direction, the optical center coordinates in the horizontal direction, and the optical center coordinates in the vertical direction, and the first parameter processing model is obtained through training through incident light set samples, the camera focal length labels in the horizontal direction corresponding to the incident light set samples, the camera focal length labels in the vertical direction corresponding to the incident light set samples, the optical center coordinate labels in the horizontal direction corresponding to the incident light set samples, and the optical center coordinate labels in the vertical direction corresponding to the incident light set samples.

11. A visual processing model training method comprising: Determining a visual data sample, a depth map label of the visual data sample, and an incident ray set label of the visual data sample, wherein the incident ray in the incident ray set label is an incident ray between a position point of a sample object in the visual data sample and a pixel point of the sample object, the position point being a point of the sample object in a three-dimensional coordinate system, and the pixel point being a point of the sample object in a two-dimensional imaging plane; Inputting the visual data sample into a visual processing model to obtain a predicted depth map and a predicted incident light set of the visual data sample; The visual processing model is trained based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

12. The visual processing model training method according to claim 11, wherein training the visual processing model based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label comprises: Training a depth map prediction unit of the visual processing model according to the predicted depth map and the depth map label; The incident light set prediction unit of the visual processing model is trained based on the predicted incident light set and the incident light set label.

13. An image processing method, comprising: Determining a target image, and inputting the target image into an image processing model; Using the image processing model, obtaining a depth map and an incident ray set of the target image, wherein the incident rays in the incident ray set are incident rays between a position point of a target object in the target image and a pixel point of the target object, the position point being a point of the target object in a three-dimensional coordinate system, and the pixel point being a point of the target object in a two-dimensional imaging plane; determining target camera parameters of a target camera model based on the set of incident rays; According to the target camera parameters and the depth map, three-dimensional point cloud data corresponding to the target image is obtained.

14. A method for training an image processing model, comprising: Determining an image sample, a depth map label of the image sample, and an incident ray set label of the image sample, wherein the incident ray in the incident ray set label is the incident ray between a position point of the sample object in the image sample and a pixel point of the sample object, the position point is a point of the sample object in a three-dimensional coordinate system, and the pixel point is a point of the sample object in a two-dimensional imaging plane; Inputting the image sample into an image processing model to obtain a predicted depth map and a predicted incident light set of the image sample; The image processing model is trained based on the predicted depth map, the depth map label, the predicted incident ray set, and the incident ray set label.

15. An image processing method, applied to traffic scenes, comprising: determining a target traffic image, and inputting the target traffic image into a traffic image processing model; Using the traffic image processing model, obtaining a depth map and an incident ray set of the target traffic image, wherein incident rays in the incident ray set are incident rays between a position point of a target object in the target traffic image and a pixel point of the target object, the position point being a point of the target object in a three-dimensional coordinate system, and the pixel point being a point of the target object in a two-dimensional imaging plane; Determining target camera parameters of a target camera model according to the incident light set, and obtaining three-dimensional point cloud data corresponding to the target traffic image according to the target camera parameters and the depth map; Determine a traffic event corresponding to the three-dimensional point cloud data according to the three-dimensional point cloud data.

16. The image processing method according to claim 15, wherein the traffic image processing model comprises a feature extraction unit, a depth map prediction unit, and an incident light set prediction unit; The method of obtaining a depth map and an incident light set of the target traffic image by using the traffic image processing model includes: Using the feature extraction unit, obtaining image features of the target traffic image; Inputting the image features into the depth map prediction unit and the incident light set prediction unit; Obtaining a depth map of the target traffic image using the depth map prediction unit; The incident light set prediction unit is used to obtain the incident light set of the target traffic image.

17. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 16 are implemented.

18. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 16.

19. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 16.

Images