Data processing method, device, electronic device and storage medium

Abstract

An embodiment of the present application provides a data processing method, device, electronic device and storage medium, the method comprising: obtaining training input data, and inputting the training input data into a depth estimation model, determining a depth estimation map of a first view, and determining a first loss function based on the depth estimation map, the training input data comprising a first view to be analyzed, at least one second view and a view pose relationship between the first view and the second view, the first view and the second view containing a target object; obtaining key points of the target object in the first view and the second view, and determining a second loss function in combination with the depth estimation map and the view pose relationship; adjusting the depth estimation model based on the first loss function and the second loss function; and being able to improve the accuracy of the depth estimation model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a data processing device, an electronic device and a storage medium. Background Art

[0002] Image depth is the basis for describing the grayscale value or color number of each pixel in the image depth information. The depth information of the scene corresponding to the image is generally described by a grayscale image of the same size. The grayscale value of each pixel in the grayscale image describes the depth value of the scene corresponding to the point. The grayscale image is also called a depth map. Only by knowing the pixel depth can the corresponding depth value of each pixel be calculated. The depth value of the image is very important for the post-processing of the image (such as special effects processing).

[0003] At present, the image depth estimation methods are mainly divided into single-view depth estimation and multi-view depth estimation. Single-view depth estimation is to analyze a single image to determine the image depth, and multi-view depth estimation is to analyze the main view to be analyzed and other views related to the main view to determine the image depth corresponding to the main view. An existing multi-view depth estimation method uses a neural network model to input the main view, other views, and the positional relationship between views into the neural network model to determine the depth estimation map of the main view, and then use the depth value of the main view manually marked to determine the loss function, and use the back-propagation algorithm to train the neural network model.

[0004] However, only manually labeled depth values ​​are used to reversely train the neural network model, and manual labeling may result in partial errors, which makes the depth estimation model less accurate and leads to low accuracy of the depth map analyzed by the model. Summary of the invention

[0005] The embodiment of the present application provides a data processing method to improve the accuracy of model analysis.

[0006] Correspondingly, an embodiment of the present application also provides a data processing device, an electronic device and a storage medium to ensure the implementation and application of the above system.

[0007] In order to solve the above problems, an embodiment of the present application discloses a data processing method, the method comprising: obtaining training input data, and inputting the training input data into a depth estimation model, determining a depth estimation map of a first view, and determining a first loss function based on the depth estimation map, the training input data including a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, the first view and the second view containing a target object; obtaining key points of the target object in the first view and the second view, and determining a second loss function in combination with the depth estimation map and the view pose relationship; adjusting the depth estimation model based on the first loss function and the second loss function.

[0008] In order to solve the above problems, an embodiment of the present application discloses a data processing method, including: obtaining input data, the input data including: a first input view to be analyzed, at least one second input view, and an input view pose relationship characterizing the positional relationship between the first input view and the second input view, the first input view and the second input view containing an object to be analyzed; inputting the input data into a trained depth estimation model to determine a depth information map of the first input view, the depth estimation model being adjusted based on a first loss function and a second loss function, the first loss function being determined based on the depth estimation map and depth marking information during the training process, and the second loss function being determined based on the positions of key points of the target object in different training views.

[0009] In order to solve the above problems, an embodiment of the present application discloses a data processing method, including: acquiring live broadcast data, and intercepting live broadcast input data from the live broadcast data, the live broadcast input data including: a first live broadcast view to be analyzed, at least one second live broadcast view, and a live broadcast view posture relationship characterizing the positional relationship between the first live broadcast view and the second live broadcast view, the first input view and the second input view contain a target object; inputting the input data into a trained depth estimation model to determine a depth information map of the first live broadcast view, the depth estimation model is adjusted based on a first loss function and a second loss function, the first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the position of the key points of the target object in different training views; based on the depth information map, performing special effects processing on the first live broadcast view to form live playback data.

[0010] In order to solve the above problems, an embodiment of the present application discloses a data processing method, including: obtaining video input data, the video input data including: first video data to be analyzed, at least one second video data, and a video pose relationship characterizing the positional relationship between the first video data and the second video data, the first video data and the second video data containing a target object; inputting the video input data into a trained depth estimation model to determine a depth information map of the first video data, the depth estimation model is adjusted based on a first loss function and a second loss function, the first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the position of the key points of the target object in different training views; based on the depth information map, performing special effects processing on the first video data to form processed video data.

[0011] In order to solve the above problems, an embodiment of the present application discloses a data processing method, including: providing a data interaction page, the data interaction page including a data upload control; based on the triggering of the data upload control, obtaining input data, and uploading the input data, the input data including: a first input view to be analyzed, at least one second input view, and an input view pose relationship characterizing the positional relationship between the first input view and the second input view, the first input view and the second input view containing an object to be analyzed; on the data interaction page, displaying a depth information map of the first input view, the depth estimation map is determined by inputting the input data into a trained depth estimation model, the depth estimation model is adjusted based on a first loss function and a second loss function, the first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the positions of key points of the target object in different training views.

[0012] In order to solve the above problems, an embodiment of the present application discloses a data processing device, including: a first function acquisition module, used to acquire training input data, and input the training input data into a depth estimation model, determine a depth estimation map of a first view, and determine a first loss function based on the depth estimation map, wherein the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, and the first view and the second view contain a target object; a second function acquisition module, used to acquire key points of the target object in the first view and the second view, and determine a second loss function in combination with the depth estimation map and the view pose relationship; an estimation model adjustment module, used to adjust the depth estimation model based on the first loss function and the second loss function.

[0013] In order to solve the above problems, an embodiment of the present application discloses a data processing device, including: an input data acquisition module, used to acquire input data, the input data including: a first input view to be analyzed, at least one second input view, and an input view pose relationship characterizing the positional relationship between the first input view and the second input view, the first input view and the second input view containing an object to be analyzed; a depth information map acquisition module, used to input the input data into a trained depth estimation model, determine the depth information map of the first input view, the depth estimation model is adjusted based on a first loss function and a second loss function, the first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the positions of key points of the target object in different training views.

[0014] In order to solve the above problems, an embodiment of the present application discloses a data processing device, including: a live data acquisition module, used to acquire live data and intercept live input data from the live data, wherein the live input data includes: a first live view to be analyzed, at least one second live view, and a live view posture relationship representing the positional relationship between the first live view and the second live view, wherein the first input view and the second input view contain a target object; a live depth map acquisition module, used to input the input data into a trained depth estimation model to determine a depth information map of the first live view, wherein the depth estimation model is adjusted based on a first loss function and a second loss function, wherein the first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the position of the key points of the target object in different training views; a live special effect processing module, used to perform special effect processing on the first live view based on the depth information map to form live playback data.

[0015] In order to solve the above problems, an embodiment of the present application discloses a data processing device, including: a video data acquisition module, used to acquire video input data, the video input data including: a first video data to be analyzed, at least one second video data, and a video pose relationship characterizing the positional relationship between the first video data and the second video data, the first video data and the second video data containing a target object; a video depth map acquisition module, used to input the video input data into a trained depth estimation model, and determine a depth information map of the first video data, the depth estimation model is adjusted based on a first loss function and a second loss function, the first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the position of the key points of the target object in different training views; a video special effects processing module, used to perform special effects processing on the first video data based on the depth information map to form processed video data.

[0016] In order to solve the above problems, an embodiment of the present application discloses a data processing device, including: an interactive page providing module, used to provide a data interaction page, the data interaction page including a data upload control; an input data determination module, used to obtain input data based on the triggering of the data upload control, and upload the input data, the input data including: a first input view to be analyzed, at least one second input view, and an input view pose relationship characterizing the positional relationship between the first input view and the second input view, the first input view and the second input view containing an object to be analyzed; a depth information map output module, used to display the depth information map of the first input view on the data interaction page, the depth estimation map is determined by inputting the input data into a trained depth estimation model, the depth estimation model is adjusted based on a first loss function and a second loss function, the first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the position of the key points of the target object in different training views.

[0017] In order to solve the above problems, an embodiment of the present application discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the above embodiments.

[0018] In order to solve the above problems, the embodiments of the present application disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the above embodiments.

[0019] Compared with the prior art, the embodiments of the present application have the following advantages:

[0020] In an embodiment of the present application, the training input data can be input into the depth estimation model to determine the depth estimation map; after determining the depth estimation map, the first loss function can be determined based on the depth marking information and the depth estimation map; then the key points of the target object in the first view and the second view can be obtained, and the second loss function can be determined in combination with the depth estimation map and the view pose relationship; then the depth estimation model can be trained based on the first loss function and the second loss function. In an embodiment of the present application, the first loss function can be determined based on the pre-labeled depth marking information and the depth estimation map, and the second loss function can be determined based on the key points of the target object in the first view and the second view. Then, based on the first loss function and the second loss function, the depth estimation model is trained through the back propagation algorithm, so that the training of the depth estimation model can rely on the labeled depth marking information and the positional relationship of the key points of the target object in different views, which reduces the dependence of the depth estimation model on the labeled depth marking information and can improve the analysis accuracy of the depth estimation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A is a schematic diagram of a data processing method according to an embodiment of the present application;

[0022] Figure 1B It is a flowchart of a data processing method according to an embodiment of the present application;

[0023] Figure 2 is a flowchart of a data processing method according to another embodiment of the present application;

[0024] Figure 3 is a flowchart of a data processing method according to another embodiment of the present application;

[0025] Figure 4 is a flowchart of a data processing method according to another embodiment of the present application;

[0026] Figure 5 is a flowchart of a data processing method according to another embodiment of the present application;

[0027] Figure 6 is a flowchart of a data processing method according to another embodiment of the present application;

[0028] Figure 7 is a flowchart of a data processing method according to another embodiment of the present application;

[0029] Figure 8 is a structural schematic diagram of a data processing device according to an embodiment of the present application;

[0030] Fig. 9 is a structural schematic diagram of a data processing device according to another embodiment of the present application;

[0031] Fig.10 is a structural schematic diagram of a data processing device according to another embodiment of the present application;

[0032] Fig.11 is a structural schematic diagram of a data processing device according to another embodiment of the present application;

[0033] Fig.12 is a structural schematic diagram of a data processing device according to another embodiment of the present application;

[0034] Fig.13 It is a schematic diagram of the structure of an exemplary device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] The embodiments of the present application can be applied to the field of depth estimation of multiple views to determine the image depth of a view (or image), wherein image depth is the basis for describing the grayscale value or color number of each pixel in the image depth information. The depth information of the scene corresponding to the image is generally described by a grayscale image of the same size. The grayscale value of each pixel in the grayscale image describes the depth value of the scene corresponding to the point. The grayscale image is also called a depth map. Only by knowing the pixel depth can the depth value corresponding to each pixel be calculated.

[0037] like Figure 1A As shown, in the embodiment of the present application, a neural network model (such as a depth estimation model) can be used to analyze the input data to obtain the corresponding image depth information (or depth map, depth information map, depth estimation map). In the training process of the depth estimation model, the embodiment of the present application can obtain the training input data and input the training input data into the depth estimation model to obtain the depth estimation map, where, Figure 1B As shown, the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, wherein the first view and the second view contain a target object, wherein the view pose relationship refers to the relationship between the position and posture (such as shooting angle) of the acquisition device corresponding to the view, and specifically, the view pose relationship between the first view and the second view is related to the position and posture of the acquisition device of the view, the position of the view acquisition device can be understood as the coordinates of the view acquisition device in three-dimensional space, and the posture of the view acquisition device can be understood as the shooting angle of the view acquisition device in three-dimensional space. After obtaining the depth estimation map, the difference between the depth tag information pre-set for the first view and the depth estimation map can be used to determine the first loss function, and the first loss function is used to adjust the depth estimation model.

[0038] After the depth estimation map is determined, the key points of the target object can be identified in the first view and the second view to determine the first position information of the key points of the target object in the first view (corresponding to the first position map) and the second position information of the key points of the target object in the second view (corresponding to the second position map), and based on the depth estimation map and the view pose relationship that characterizes the positional relationship between the first view and the second view, the first position information is mapped to the second view to obtain the mapping position information of the key points of the target object in the second view (corresponding to the mapping position map), and then the position difference between the mapping position information and the second position information is analyzed to obtain the second loss function, which is used to adjust the depth estimation model. For example, in the case of Figure 1A In the example shown, key point positions can be identified for the first view and the second view to determine positions 1 and 2 of target object 1 and target object 2 in the first view, and positions 3 and 4 of target object 1 and target object 2 in the second view. Then, based on the view position relationship and the depth estimation map, positions 1 and 2 in the first position map can be mapped to the mapping position map to obtain positions 5 and 6. Then, based on the difference between positions 3 and 5 and the difference between positions 4 and 6, a second loss function can be determined, and the depth estimation model can be adjusted using the second loss function. It should be noted that Figure 1A The first position map, the second position map and the mapping position map are only for the convenience of understanding. In the actual data processing process, the coordinates of the key points of the target object in the view can be analyzed, and the coordinates and the view pose relationship can be used for mapping conversion. After determining the first loss function and the second loss function, the embodiment of the present application can adjust the depth estimation model according to the first loss function and the second loss function.

[0039] The existing method of depth recognition of images is to use manually annotated depth mark information to determine the first loss function, and train the depth estimation model based on the first loss function. However, in this way, the depth estimation model is too dependent on the manually annotated depth mark information, and the manual marking may have partial errors, which makes the accuracy of the depth estimation model low. In the embodiment of the present application, the first loss function can be determined based on the pre-annotated depth mark information and the depth estimation map, and the second loss function can be determined based on the key points of the target object in the first view and the second view. Then, based on the first loss function and the second loss function, the depth estimation model is adjusted through the back propagation algorithm, so that the training of the depth estimation model can rely on the depth mark information and the positional relationship between the positions of the key points of the target object in the first view and the second view, reducing the dependence of the depth estimation model on the annotated depth mark information, and improving the analysis accuracy of the depth estimation model.

[0040] The embodiments of the present application can be applied to the analysis and processing scenes of data such as videos and images to analyze the depth information of the target object, wherein the target object in the view may include people, animals, moving targets, scenery, and other things, such as golf balls, footballs, vehicles, buildings, etc. For example, the embodiments of the present application can be applied to live broadcast scenes, and can perform in-depth analysis on live broadcast videos, determine the corresponding depth information graph, and add special effects to the live broadcast videos according to the depth information graph, such as adding special effects to the anchor according to augmented reality (AR) technology, such as adding glasses, cat ears and other virtual information to the anchor, so as to achieve the combination of virtual and reality and improve the live broadcast effect. For another example, the embodiments of the present application can also be applied to the scenes of sports events and variety shows, such as performing in-depth analysis on sports event videos (such as live broadcast videos of events), and using the depth information graph obtained by the analysis, performing special effects processing (such as slow motion, close-up, etc.) on the live broadcast videos of events to form playback data for event playback. For another example, the embodiment of the present application can also be applied to the scene of e-commerce selling goods, and can perform in-depth analysis on the live video of e-commerce selling goods, determine the depth information of the target object (such as goods or people), and optimize the target object according to the depth information, such as adding prices and product descriptions around the goods. For another example, the embodiment of the present application can also be applied to the scene of identifying advertising videos, and can perform in-depth analysis on the goods in the advertising videos, determine the depth information of the goods, and perform semantic recognition on the goods according to the depth information of the goods, so as to determine the detailed information of the goods in the advertisement, and display it to the user, so as to facilitate the user to understand the characteristics of the goods in the advertisement. For another example, the embodiment of the present application can also be applied to the data processing scene of traffic monitoring video, security monitoring video, and variety show video, perform in-depth analysis on the video data, and use the depth information map obtained by analysis to perform object recognition (such as face recognition, license plate number recognition, etc.) on the target object. For another example, the embodiment of the present application can also be applied to the data processing scene of image synthesis, and can perform in-depth analysis on multiple related images, and use the depth information map obtained by analysis to synthesize multiple images into a target image, so that the target image contains more content. The embodiments of the present application may also be applied to scenarios where depth analysis is performed on videos or images, and a three-dimensional stereo model is constructed using a depth information map.

[0041] The present application also provides a data processing method, which can be applied to a processing end, and the processing end can be understood as a computing device for training a depth estimation model. ... Figure 2 As shown, the method includes:

[0042] Step 202, obtain training input data, and input the training input data into the depth estimation model, determine the depth estimation map of the first view, and determine the first loss function according to the depth estimation map, wherein the training input data includes the first view to be analyzed, at least one second view, and the view pose relationship between the first view and the second view, wherein the first view and the second view contain the target object. Before adjusting the depth estimation model, the processing end may receive the set training input data and the corresponding depth tag information (the analysis result corresponding to the training input data, or the annotation), wherein the training input data may include the first view, the second view, and the view pose relationship between the views, the first view is the view to be analyzed, the depth estimation model may be understood as an algorithm model, which can analyze the input data, determine the depth information of each target object in the input data, and form a depth estimation map. In this embodiment, the depth estimation model is used to analyze the depth estimation map corresponding to the first view. The second view is a view related to the first view, such as an adjacent view collected by a collection device adjacent to the collection device that shoots the first view, a relative view collected by a collection device opposite to the collection device that shoots the first view, etc. The view pose relationship can be determined based on the positional relationship between the collection devices of the first view and the second view. The depth marking information can be determined by manual labeling or other methods, which can be set according to needs.

[0043] After determining the training input data, the processing end can input the training input data into the depth estimation model to determine the depth estimation map of the first view, so as to determine the first loss function based on the depth estimation map. In order to distinguish the training stage and the application stage of the depth estimation model, the analysis result of the depth estimation model in the training stage is referred to as a depth estimation map in the embodiment of the present application, and the analysis result of the trained depth estimation model in the application stage is referred to as a depth information map. The processing end inputs the training input data into the depth estimation model, and the depth estimation model can analyze the first view and the second view to obtain the image features of the view, and then use the view pose relationship between the first view and the second view to adjust the image features of the first view to obtain the depth estimation map. Specifically, as an optional embodiment, the depth estimation model is used to perform the following steps: obtain the first image features of the first view and the second image features of the second view; determine the reference image features of the second view mapped to the first view based on the view pose relationship and the second image features; adjust the first image features based on the reference image features to obtain the depth estimation map of the first view.

[0044] In one example, the depth estimation map may include depth information of each object in the view (which may be represented by a grayscale value), and each object in the view may include a target object, a background, other objects, etc. For example, the objects in the view may include a person (target object), grass (background), and trees (other objects). In another example, the depth estimation map may also include depth information of the target object in the view, and the target object in the view may be one or more, which may be set according to the requirements. The depth estimation model may analyze the target object in the view to obtain the depth information of the target object as an image feature. Afterwards, the second image feature of the second view may be mapped to the first view in combination with the view pose relationship between the first view and the second view to obtain a reference image feature, and then the first image feature of the first view may be adjusted according to the difference between the reference feature and the first image feature to obtain a depth estimation map. For example, the depth value corresponding to the target object in the first image feature may be 5.3 meters, and the depth values ​​corresponding to the target object in the reference image feature of the first view mapped by the two second image features are 5 meters and 5.1 meters respectively. Then, the depth value of the target object in the first image feature may be reduced according to the two reference image features, thereby obtaining a depth estimation map.

[0045] After determining the depth estimation map, the processing end may determine a first loss function based on the depth estimation map and the depth tag information of the first view. Specifically, in an optional embodiment, determining the first loss function based on the depth estimation map includes: determining the first loss function based on the depth estimation map and the depth tag information of the first view. The depth tag information of the first view includes the actual depth value set for the target object. The processing end may determine the first loss function based on the difference between the estimated depth value of the target object in the depth estimation map and the actual depth value set for the target object, so as to adjust the depth estimation model through the first loss function.

[0046] In addition to reversely adjusting the depth estimation model according to the actual depth value set for the target pair, the embodiment of the present application can also obtain the key points of the target object in the first view and the second view in step 204, and determine the second loss function in combination with the depth estimation map and the view pose relationship. The key points of the target object are points with the same semantics in the first view and the second view. The key points of the target object can be pre-set points. For example, for a person, one or more of the person's head, arm, elbow, leg, knee, and foot can be used as key points, and the first view and the second view are identified to determine the position (such as coordinates) of the key points of the target object in the view, and then combine the depth information of the target object in the depth estimation map and the view pose relationship between the views to determine the error of the depth estimation information of the target object, and then obtain the second loss function, so as to adjust the depth estimation model according to the second loss function.

[0047] As an optional embodiment, the first loss function includes a supervised loss function, and the second loss function includes an unsupervised loss function. The supervised loss function is obtained through supervised learning, which is a machine learning task that infers a function from a labeled training data set. The unsupervised loss function is obtained through unsupervised learning, which refers to solving various problems in model recognition based on training samples with unknown categories (not labeled).

[0048] After determining the first loss function and the second loss function, the depth estimation model can be adjusted in step 206 according to the first loss function and the second loss function. The depth estimation model can be trained by back propagation according to the first loss function and the second loss function. Back propagation training is training using a back propagation algorithm. The back propagation algorithm (BP algorithm) is a learning algorithm suitable for multi-layer neural networks, which is based on the gradient descent method. The back propagation algorithm mainly consists of two links (excitation propagation and weight update) that are repeatedly iterated until the network's response to the input reaches a predetermined target range.

[0049] The embodiments of the present application can reversely adjust the depth estimation model based on the first loss function and the second loss function, so that the training of the depth estimation model can rely on the positional relationship between the positions of the key points of the target object in the first view and the second view for training, thereby reducing the dependence of the depth estimation model on the annotated depth mark information, and improving the analysis accuracy of the depth estimation model.

[0050] The processing end can map the key points in the first view to the second view based on the depth information of the target object in the depth estimation map and the view pose relationship between the views, and then determine the error of the depth information of the target object in the depth estimation map based on the position of the key point mapping and the position of the key point in the second view, and determine the second loss function. Specifically, as an optional embodiment, the key points of the target object in the first view and the second view are obtained, and the second loss function is determined in combination with the depth estimation map and the view pose relationship, including: obtaining the first position information of the key points in the first view and the second position information of the key points in the second view; based on the depth estimation map and the view pose relationship, determining the mapping position information of the first position information mapped to the second view; and determining the second loss function based on the second position information and the mapping position information. The processing end can perform image recognition on the first view and the second view, determine the target object in the first view and the second view, and then determine the first position information of the key points of the target object in the first view and the second position information of the key points of the target object in the second view. Among them, image recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. After determining the first position information and the second position information of the key points of the target object, the first position information in the first view can be mapped to the second view based on the depth information of the target object in the depth estimation map and the view pose relationship between the views to determine the mapping position information. Then, based on the difference between the actual second position information of the key points of the target object in the second view and the mapped position information (related to the accuracy of the depth estimation map), a second loss function is obtained, so that the depth estimation model can be reversely adjusted using the second loss function.

[0051] The processing end can map the first position information to the three-dimensional space of the camera coordinate system according to the depth estimation map and the view pose relationship, and then determine the mapping position information. Specifically, as an optional embodiment, the mapping position information of the first position information to the second view based on the depth estimation map and the view pose relationship includes: determining the first depth information corresponding to the first position information according to the depth estimation map; determining the second depth information of the key point in the second view according to the view pose relationship and the first depth information; and determining the mapping position information of the key point of the second view according to the second depth information. The processing end can determine the first depth information of each key point of the target object according to the depth estimation map and the first position information of the key point of the target object, and then map the key point of the target object to the three-dimensional space of the camera coordinate system of the shooting view according to the first depth information, and perform mapping between different camera coordinate systems according to the view pose relationship, determine the second depth information of the key point of the target object mapped to the second view, and then determine the mapping position information, so as to determine the difference between the mapping position and the second position information, and determine the second loss function.

[0052] After the depth estimation model is trained in the embodiment of the present application, the trained depth estimation model can be deployed in the processing end to analyze the input data, or the depth estimation model can be deployed in other devices for analysis. The specific deployment can be based on the actual application scenario. For example, the trained depth estimation model can be deployed in a server that transfers and stores live data, or the trained depth estimation model can be deployed in a terminal device that performs image synthesis.

[0053] The trained depth estimation model can analyze the input multiple views to determine the depth information map of the first view to be analyzed. Specifically, as an optional embodiment, the method further includes: obtaining input data, the input data including: the first input view to be analyzed, at least one second input view, and the input view pose relationship that characterizes the positional relationship between the first input view and the second input view, the first input view and the second input view containing the target object; inputting the input data into the trained depth estimation model to determine the depth information map of the first view. After the depth estimation model is trained, the first input view to be analyzed, at least one second input view, and the input view pose relationship can be input into the trained depth estimation model, the trained depth estimation model analyzes the first image feature of the first input view and the second image feature of the second input view, and then determines the reference image feature of the first input view mapped to the second image feature according to the input view pose relationship, and adjusts the first image feature according to the reference image feature, thereby obtaining the depth information map of the first input view, the depth information map can be understood as the depth estimation map of the depth estimation model during the training process.

[0054] After determining the depth information map of the first view, corresponding image processing can be performed on the first view according to the depth information contained in the depth information map, such as adding special effects, multi-image synthesis, image scene recognition, and building a three-dimensional model. Specifically, as an optional embodiment, the method also includes: performing image processing on the first view based on the depth information map, and the image processing includes special effects processing, image synthesis processing, image scene recognition, and three-dimensional modeling processing.

[0055] In an embodiment of the present application, the training input data can be input into the depth estimation model to determine the depth estimation map; after determining the depth estimation map, the first loss function can be determined based on the depth tag information and the depth estimation map; then the key points of the target object in the first view and the second view can be obtained, and the second loss function can be determined in combination with the depth estimation map and the view pose relationship; then the depth estimation model can be trained based on the first loss function and the second loss function. In an embodiment of the present application, the first loss function can be determined based on the pre-annotated depth tag information and the depth estimation map, and the second loss function can be determined based on the key points of the target object in the first view and the second view. Then, based on the first loss function and the second loss function, the depth estimation model is adjusted through the back propagation algorithm, so that the training of the depth estimation model can be based on the positional relationship between the positions of the key points of the target object in the first view and the second view, which reduces the dependence of the depth estimation model on the annotated depth tag information and can improve the analysis accuracy of the depth estimation model.

[0056] On the basis of the above embodiments, the embodiment of the present application further provides a data processing method, which can be executed by a processing end, and the method includes:

[0057] Step 302: Acquire training input data, where the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, where the first view and the second view contain a target object.

[0058] Step 304: Input the training input data into the depth estimation model to determine the depth estimation map of the first view. As an optional embodiment, the depth estimation model is used to perform the following steps: obtain the first image features of the first view and the second image features of the second view; determine the reference image features of the second view mapped to the first view based on the view pose relationship and the second image features; adjust the first image features based on the reference image features to obtain the depth estimation map of the first view.

[0059] Step 306: Determine a first loss function according to the depth estimation map and the depth tag information of the first view.

[0060] Step 308: Obtain first position information of the key point in the first view and second position information of the key point in the second view.

[0061] Step 310: Determine first depth information corresponding to the first position information according to the depth estimation map;

[0062] Step 312: Determine second depth information of key points in a second view according to the view pose relationship and the first depth information.

[0063] Step 314: Determine mapping position information of key points of the second view according to the second depth information.

[0064] Step 316: Determine a second loss function based on the second position information and the mapping position information.

[0065] Step 318: Adjust the depth estimation model according to the first loss function and the second loss function. As an optional embodiment, the first loss function includes a supervised loss function, and the second loss function includes an unsupervised loss function.

[0066] Step 320: Obtain input data, where the input data includes: a first input view to be analyzed, at least one second input view, and an input view pose relationship characterizing a positional relationship between the first input view and the second input view, where the first input view and the second input view contain a target object.

[0067] Step 322: Input the input data into the trained depth estimation model to determine the depth information map of the first view. It should be noted that the embodiment of the present application takes the deployment of the trained depth estimation model in the processing end for data processing as an example to describe the trained depth estimation model. The trained depth estimation model in the embodiment of the present application can also be deployed in other devices for data processing.

[0068] Step 324: perform image processing on the first view according to the depth information map, wherein the image processing includes at least one of special effect processing, image synthesis processing, image scene recognition and three-dimensional modeling processing.

[0069] In an embodiment of the present application, training input data can be input into a depth estimation model to determine a depth estimation map; after determining the depth estimation map, a first loss function can be determined based on the depth marking information and the depth estimation map; then, first position information of a key point of the target object in the first view and second position information of the key point of the target object in the second view can be obtained, and then, based on the depth estimation map, first depth information corresponding to the first position information can be determined, and based on the view pose relationship, the first depth information can be mapped to the second depth information of the second view, and a mapping position relationship of the key point of the target object in the first view to the second view can be determined based on the second depth information, and the difference between the mapping position relationship and the second position relationship can be analyzed to determine the second loss function; then, the depth estimation model can be adjusted based on the first loss function and the second loss function. After the depth estimation model is trained, the first input view to be analyzed, at least one second input view and the input view pose relationship can be input into the depth estimation model. The depth estimation model analyzes the first image features of the first input view and the second image features of the second input view, and then determines the reference image features of the first input view mapped with the second image features based on the input view pose relationship, and adjusts the first image features based on the reference image features to obtain the depth information map of the first input view.

[0070] On the basis of the above embodiments, the embodiments of the present application also provide a data processing method, which can be executed by a processing device. The processing device can be a server for data transfer and storage, or a terminal device for image synthesis, adding special effects, and scene recognition. This embodiment can analyze the input multiple views (a first input view and at least one second input view) through a trained depth estimation model, determine the depth information map corresponding to the first input view, and output it. Figure 4 As shown, the method includes:

[0071] Step 402: Obtain input data, where the input data includes: a first input view to be analyzed, at least one second input view, and an input view pose relationship characterizing a positional relationship between the first input view and the second input view, where the first input view and the second input view contain an object to be analyzed.

[0072] Step 404: input the input data into the trained depth estimation model to determine the depth information map of the first input view, wherein the depth estimation model is adjusted based on a first loss function and a second loss function, wherein the first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the positions of the key points of the target object in different training views.

[0073] The implementation of the embodiment of the present application is similar to the implementation of the above method embodiment. The specific implementation can refer to the specific implementation of the above method embodiment, which will not be repeated here.

[0074] In an embodiment of the present application, during the training process of the depth estimation model, a first loss function can be determined based on the difference between the analysis result (depth estimation map) during the training process and the depth tag information; a second loss function can be determined based on the position of the target object in different training views, combined with the depth estimation map and the view pose relationship between different training views; then, based on the first loss function and the second loss function, the depth estimation model is adjusted, which reduces the dependence of the depth estimation model on the annotated depth tag information, and can improve the analysis accuracy of the depth estimation model. After the training of the depth estimation model is completed, the first input view to be analyzed, at least one second input view and the input view pose relationship can be input into the depth estimation model, the depth estimation model analyzes the first image features of the first input view and the second image features of the second input view, and then determines the reference image features of the first input view mapped to the second image features based on the input view pose relationship, and adjusts the first image features based on the reference image features, thereby obtaining a depth information map of the first input view, and the depth information map can be understood as the depth estimation map of the depth estimation model during the training process.

[0075] On the basis of the above embodiments, the embodiments of the present application also provide a data processing method, which can be applied in the scenario of processing live broadcast data. The method can be executed by a processing device, which can be a server for transferring and storing live broadcast data. The method can intercept the live broadcast data to obtain live broadcast input data, and perform depth analysis on multiple views (a first input view and at least one second input view) in the live broadcast input data through a trained depth estimation model, determine the corresponding depth information map, and add special effects to the live broadcast input data based on the depth information map. Figure 5 As shown, the method includes:

[0076] Step 502: Acquire live broadcast data, and intercept live broadcast input data from the live broadcast data, wherein the live broadcast input data includes: a first live broadcast view to be analyzed, at least one second live broadcast view, and a live broadcast view posture relationship representing a positional relationship between the first live broadcast view and the second live broadcast view, and the first input view and the second input view contain a target object.

[0077] Step 504: input the input data into the trained depth estimation model to determine the depth information map of the first live view. The depth estimation model is adjusted based on a first loss function and a second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process. The second loss function is determined based on the positions of the key points of the target object in different training views.

[0078] Step 506: Perform special effects processing on the first live view according to the depth information map to form live playback data.

[0079] The implementation of the embodiment of the present application is similar to the implementation of the above method embodiment. The specific implementation can refer to the specific implementation of the above method embodiment, which will not be repeated here.

[0080] In an embodiment of the present application, during the training process of the depth estimation model, a first loss function can be determined based on the difference between the analysis results (depth estimation map) during the training process and the depth mark information; a second loss function can be determined based on the position of the target object in different training views (views in the training data, such as the first view and the second view), combined with the depth estimation map and the view pose relationship between different training views; and the depth estimation model is then adjusted based on the first loss function and the second loss function, thereby reducing the dependence of the depth estimation model on the annotated depth mark information and improving the analysis accuracy of the depth estimation model. After the depth estimation model is trained, it can be applied to the scene of analyzing the relevant data of the live broadcast, such as live broadcast of sports events, online live broadcast, etc. Some wonderful pictures that need to be replayed can be intercepted from the live broadcast data as live broadcast input data, and then the live broadcast input data is input into the depth estimation model. The depth estimation model analyzes the first image features of the first live broadcast view and the second image features of the second live broadcast view, and then determines the second image features mapped to the reference image features of the first live broadcast view based on the posture relationship of the live broadcast view, and adjusts the first image features based on the reference image features, thereby obtaining the depth information map of the first live broadcast view. The depth information map can be understood as the depth estimation map of the depth estimation model. Afterwards, according to the depth information of each object in the depth estimation map, special effects (such as slow motion, magnification and other special effects processing) can be added to the first live broadcast view, a three-dimensional scene model can be constructed, scene recognition, image synthesis and other processing can be performed.

[0081] On the basis of the above-mentioned embodiments, the embodiments of the present application also provide a data processing method, which can be applied in the scenario of processing video data. The method can be executed by a processing device, and the processing device can be a server for data transfer and storage, or a terminal device for adding special effects. In this embodiment, the video input data can be input into a trained depth estimation model, and the trained depth estimation model can be used to perform depth analysis on multiple views (first video data and at least one second video data) in the video input data to determine the corresponding depth information map, and based on the depth information map, add special effects to the video input data. Figure 6 As shown, the method includes:

[0082] Step 602: Obtain video input data, wherein the video input data includes: first video data to be analyzed, at least one second video data, and a video pose relationship representing a positional relationship between the first video data and the second video data, wherein the first video data and the second video data contain a target object.

[0083] Step 604: input the video input data into the trained depth estimation model to determine the depth information map of the first video data. The depth estimation model is adjusted based on a first loss function and a second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process. The second loss function is determined based on the position of the key points of the target object in different training views.

[0084] Step 606: Perform special effects processing on the first video data according to the depth information map to form processed video data.

[0085] The implementation of the embodiment of the present application is similar to the implementation of the above method embodiment. The specific implementation can refer to the specific implementation of the above method embodiment, which will not be repeated here.

[0086] In the embodiment of the present application, during the training process of the depth estimation model, the first loss function can be determined according to the difference between the analysis result (depth estimation map) during the training process and the depth labeling information; according to the position of the target object in different training views, the second loss function is determined in combination with the depth estimation map and the view pose relationship between different training views; then, according to the first loss function and the second loss function, the depth estimation model is adjusted, which reduces the dependence of the depth estimation model on the annotated depth labeling information, and can improve the analysis accuracy of the depth estimation model. After the depth estimation model is trained, it can be applied in the scene of analyzing the video input data, such as the scene of analyzing the video data such as the traffic monitoring video and the security monitoring video, the video input data can be input into the depth estimation model, the depth estimation model analyzes the first image feature of the first video data and the second image feature of the second video data, and then according to the video pose relationship, determines the second image feature to be mapped to the reference image feature of the first video data, and adjusts the first image feature according to the reference image feature, thereby obtaining the depth information map of the first video data, and the depth information map can be understood as the depth estimation map of the depth estimation model. Afterwards, special effects may be added to the first video data, a three-dimensional scene model may be constructed, scene recognition may be performed, and image synthesis may be performed, etc., based on the depth information of each object in the depth estimation image.

[0087] On the basis of the above embodiments, the embodiments of the present application also provide a data processing method, which can be applied to a user's terminal, and can provide a data interaction page on the terminal. The user can upload input data through the data interaction page. The processing device can analyze the input multiple views (a first input view and at least one second input view) through a trained depth estimation model, determine the depth information map corresponding to the first input view, and send it to the terminal for display on the data interaction page of the terminal. Specifically, Figure 7 As shown, the method includes:

[0088] Step 702: Provide a data interaction page, wherein the data interaction page includes a data upload control.

[0089] Step 704: based on triggering the data upload control, obtain input data and upload the input data, wherein the input data includes: a first input view to be analyzed, at least one second input view, and an input view posture relationship characterizing a positional relationship between the first input view and the second input view, wherein the first input view and the second input view contain an object to be analyzed.

[0090] Step 706: On the data interaction page, display a depth information map of the first input view, wherein the depth estimation map is determined by inputting the input data into a trained depth estimation model, and the depth estimation model is adjusted based on a first loss function and a second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the positions of key points of the target object in different training views.

[0091] The implementation of the embodiment of the present application is similar to the implementation of the above method embodiment. The specific implementation can refer to the specific implementation of the above method embodiment, which will not be repeated here.

[0092] In an embodiment of the present application, a data interaction page can be provided in the terminal, and the user can upload input data by triggering the data upload control. The processing device can use the trained depth estimation model to analyze the first image features of the first input view and the second image features of the second input view, and then determine the reference image features of the first input view mapped to the second image features based on the input view posture relationship, and adjust the first image features based on the reference image features, thereby obtaining a depth information map of the first input view. The depth information map can be understood as a depth estimation map of the depth estimation model during the training process. After determining the depth information map, the processing device can send the depth information map to the terminal, and the terminal outputs the depth information map through the data interaction page, such as displaying the depth information map in the data interaction page, or providing a download link on the data interaction page so that the user can download the depth information map.

[0093] On the basis of the above embodiments, the embodiments of the present application further provide an object recognition method, which can be applied to data processing scenarios of traffic monitoring videos, security monitoring videos, and variety show videos, and can perform in-depth analysis on video data, and use the depth information map obtained by the analysis to perform object recognition on the target object. Specifically, the method includes:

[0094] Video input data is acquired, wherein the video input data includes: first video data to be analyzed, at least one second video data, and a video posture relationship representing a positional relationship between the first video data and the second video data, wherein the first video data and the second video data contain a target object.

[0095] The video input data is input into a trained depth estimation model to determine a depth information map of the first video data. The depth estimation model is adjusted based on a first loss function and a second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process. The second loss function is determined based on the positions of key points of the target object in different training views.

[0096] According to the depth information map, object recognition is performed on the target object in the first video data to determine an object recognition result.

[0097] The implementation of the embodiment of the present application is similar to the implementation of the above method embodiment. The specific implementation can refer to the specific implementation of the above method embodiment, which will not be repeated here.

[0098] In the embodiment of the present application, the depth estimation model can be adjusted and trained according to the first loss function and the second loss function. The trained depth estimation model can be applied to data processing scenarios of video data such as traffic monitoring videos, security monitoring videos, variety show videos, etc. The trained depth estimation model can be used to analyze the relevant video data to determine the depth information map of the video data, and then the target object in the video data can be identified based on the depth information map to determine the semantics of the target object. For example, in traffic monitoring videos, the target object can be identified based on the depth information map to determine that the target object is a tree or a vehicle, and then the license plate, speed and other information of the vehicle can be determined. For another example, it can be applied in the field of security monitoring videos, and the target object can be identified based on the depth information map to determine that the target object is a person, an animal, etc., so as to perform face recognition on the person and determine the person's authority and other information.

[0099] On the basis of the above embodiments, the embodiments of the present application further provide an image synthesis method, which can be applied in a data processing scenario of image synthesis, can perform depth analysis on multiple related images, and use the depth information map obtained by the analysis to fuse the multiple images into a target image, so that the target image contains more content. Specifically, the method includes:

[0100] Image input data is acquired, wherein the image input data includes: a first image to be analyzed, at least one second image, and an image posture relationship representing a positional relationship between the first image and the second image, wherein the first image and the second image contain a target object.

[0101] The image input data is input into the trained depth estimation model to determine the depth information map. The depth estimation model is adjusted based on a first loss function and a second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process. The second loss function is determined based on the positions of the key points of the target object in different training views.

[0102] According to the depth information map, the first image and the second image are fused to obtain a target image.

[0103] The implementation of the embodiment of the present application is similar to the implementation of the above method embodiment. The specific implementation can refer to the specific implementation of the above method embodiment, which will not be repeated here.

[0104] In an embodiment of the present application, the depth estimation model can be adjusted and trained based on the first loss function and the second loss function. The trained depth estimation model can be applied in the scene of image synthesis, and the trained depth estimation model can be used to determine the depth information map of each image, and the depth information map of the image can include the depth information of each target object in the image. After that, the depth information of the target object in the depth information map can be used to determine the scale and direction of each image, and each image can be scaled, adjusted in direction, and cropped, and then different images can be fused to obtain a target image, so that the target image can contain the content of multiple images.

[0105] On the basis of the above embodiments, the embodiments of the present application further provide a method for constructing a three-dimensional model, which can be applied in a scene where a three-dimensional stereo model is constructed based on a video or an image, and can perform depth analysis on the video or image, and use a depth information map to construct a three-dimensional stereo model. Specifically, the method includes:

[0106] Input data is obtained, the input data including: a first input view to be analyzed, at least one second input view, and an input view pose relationship representing a positional relationship between the first input view and the second input view, the first input view and the second input view containing a target object. The first input view and the second input view can be understood as images or videos.

[0107] The input data is input into the trained depth estimation model to determine the depth information map. The depth estimation model is adjusted based on a first loss function and a second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process. The second loss function is determined based on the positions of the key points of the target object in different training views.

[0108] A three-dimensional model is constructed according to the depth information map.

[0109] The implementation of the embodiment of the present application is similar to the implementation of the above method embodiment. The specific implementation can refer to the specific implementation of the above method embodiment, which will not be repeated here.

[0110] In an embodiment of the present application, the depth estimation model can be adjusted and trained based on the first loss function and the second loss function. The trained depth estimation model can be applied in the scene of constructing a three-dimensional stereo model. The embodiment of the present application can use the trained depth estimation model to perform depth analysis on the input view to determine the corresponding depth information map, which includes the depth information of each target object. The present application can use the depth information map to construct a three-dimensional stereo model. For example, the embodiment of the present application can use the depth information of the target object in each view to determine the position of each target object in the three-dimensional space, and then merge the target objects in different views into the same three-dimensional stereo model. For another example, the embodiment of the present application can use the depth information map of the first input view to determine the depth information of the target object, and then determine the relative position of the target object relative to the constructed three-dimensional model, and then add the target object to the constructed three-dimensional model.

[0111] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0112] Based on the above embodiment, this embodiment further provides a data processing device, referring to Figure 8 , specifically it can include the following modules:

[0113] The first function acquisition module 802 is used to acquire training input data and input the training input data into the depth estimation model to determine the depth estimation map of the first view, so as to determine the first loss function based on the depth estimation map. The training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, and the first view and the second view contain a target object.

[0114] The second function acquisition module 804 is used to acquire key points of the target object in the first view and the second view, and determine the second loss function in combination with the depth estimation map and the view pose relationship.

[0115] The estimation model training module 806 is used to train the depth estimation model according to the first loss function and the second loss function.

[0116] In summary, in an embodiment of the present application, the training input data can be input into the depth estimation model to determine the depth estimation map; after determining the depth estimation map, the first loss function can be determined based on the depth marking information and the depth estimation map; then the key points of the target object in the first view and the second view can be obtained, and the second loss function can be determined in combination with the depth estimation map and the view pose relationship; then the depth estimation model can be trained based on the first loss function and the second loss function. In an embodiment of the present application, the first loss function can be determined based on the pre-labeled depth marking information and the depth estimation map, and the second loss function can be determined based on the key points of the target object in the first view and the second view. Then, based on the first loss function and the second loss function, the depth estimation model is trained through the back propagation algorithm, so that the training of the depth estimation model can rely on the labeled depth marking information and the positional relationship of the key points of the target object in different views, which reduces the dependence of the depth estimation model on the labeled depth marking information and can improve the analysis accuracy of the depth estimation model.

[0117] On the basis of the above embodiment, this embodiment further provides a data processing device, which may specifically include the following modules:

[0118] The training input data acquisition processing module is used to acquire training input data, wherein the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, wherein the first view and the second view contain a target object.

[0119] The training input data input processing module is used to input the training input data into the depth estimation model to determine the depth estimation map of the first view. As an optional embodiment, the depth estimation model is used to perform the following steps: obtaining the first image features of the first view and the second image features of the second view; determining the reference image features of the second view mapped to the first view based on the view pose relationship and the second image features; adjusting the first image features based on the reference image features to obtain the depth estimation map of the first view.

[0120] The first loss function acquisition processing module is used to determine a first loss function according to the depth estimation map and the depth marking information of the first view.

[0121] The object key point positioning processing module is used to obtain first position information of the key point in the first view and second position information of the key point in the second view.

[0122] A first depth information acquisition processing module, used to determine first depth information corresponding to the first position information according to the depth estimation map;

[0123] The second depth information acquisition processing module is used to determine the second depth information of the key point in the second view according to the view posture relationship and the first depth information.

[0124] The mapping position information acquisition processing module is used to determine the mapping position information of the key point of the second view according to the second depth information.

[0125] The second loss function acquisition processing module is used to determine the second loss function according to the second position information and the mapping position information.

[0126] The depth estimation model training processing module is used to adjust the depth estimation model according to the first loss function and the second loss function. As an optional embodiment, the first loss function includes a supervised loss function, and the second loss function includes an unsupervised loss function.

[0127] An input data acquisition processing module is used to acquire input data, wherein the input data includes: a first input view to be analyzed, at least one second input view, and an input view posture relationship that characterizes the positional relationship between the first input view and the second input view, wherein the first input view and the second input view contain a target object.

[0128] The input data input processing module is used to input the input data into the trained depth estimation model to determine the depth information map of the first view.

[0129] A data image processing module is used to perform image processing on the first view according to the depth information map, wherein the image processing includes at least one of special effect processing, image synthesis processing, image scene recognition and three-dimensional modeling processing.

[0130] In summary, in an embodiment of the present application, the training input data can be input into the depth estimation model to determine a depth estimation map; after determining the depth estimation map, the first loss function can be determined based on the depth marking information and the depth estimation map; then the first position information of the key point of the target object in the first view and the second position information of the key point of the target object in the second view can be obtained, and then the first depth information corresponding to the first position information can be determined based on the depth estimation map, and the first depth information is mapped to the second depth information of the second view based on the view posture relationship, and the mapping position relationship of the key point of the target object in the first view to the second view is determined based on the second depth information, and the difference between the mapping position relationship and the second position relationship is analyzed to determine the second loss function; then the depth estimation model can be adjusted based on the first loss function and the second loss function. After the depth estimation model is trained, the first input view to be analyzed, at least one second input view and the input view pose relationship can be input into the depth estimation model. The depth estimation model analyzes the first image features of the first input view and the second image features of the second input view, and then determines the reference image features of the first input view mapped with the second image features based on the input view pose relationship, and adjusts the first image features based on the reference image features to obtain the depth information map of the first input view.

[0131] Based on the above embodiment, this embodiment further provides a data processing device, referring to Fig. 9 , specifically it can include the following modules:

[0132] The input data acquisition module 902 is used to acquire input data, wherein the input data includes: a first input view to be analyzed, at least one second input view, and an input view posture relationship characterizing a positional relationship between the first input view and the second input view, wherein the first input view and the second input view contain an object to be analyzed.

[0133] The depth information map acquisition module 904 is used to input the input data into the trained depth estimation model to determine the depth information map of the first input view. The depth estimation model is adjusted based on the first loss function and the second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the positions of the key points of the target object in different training views.

[0134] In summary, in the embodiment of the present application, during the training process of the depth estimation model, the first loss function can be determined based on the difference between the analysis result (depth estimation map) during the training process and the depth mark information; the second loss function can be determined based on the position of the target object in different training views, combined with the depth estimation map and the view pose relationship between different training views; then the depth estimation model is trained based on the first loss function and the second loss function, which reduces the dependence of the depth estimation model on the annotated depth mark information and can improve the analysis accuracy of the depth estimation model. After the training of the depth estimation model is completed, the first input view to be analyzed, at least one second input view and the input view pose relationship can be input into the depth estimation model, the depth estimation model analyzes the first image features of the first input view and the second image features of the second input view, and then determines the second image features mapped to the reference image features of the first input view based on the input view pose relationship, and adjusts the first image features based on the reference image features, thereby obtaining the depth information map of the first input view, and the depth information map can be understood as the depth estimation map of the depth estimation model during the training process.

[0135] Based on the above embodiment, this embodiment further provides a data processing device, referring to Fig.10 , specifically it can include the following modules:

[0136] The live data acquisition module 1002 is used to acquire live data and intercept live input data from the live data, wherein the live input data includes: a first live view to be analyzed, at least one second live view, and a live view posture relationship representing a positional relationship between the first live view and the second live view, wherein the first input view and the second input view contain a target object.

[0137] The live depth map acquisition module 1004 is used to input the input data into the trained depth estimation model to determine the depth information map of the first live view. The depth estimation model is adjusted based on the first loss function and the second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the position of the key points of the target object in different training views.

[0138] The live special effect processing module 1006 is used to perform special effect processing on the first live view according to the depth information map to form live playback data.

[0139] In summary, in the embodiments of the present application, during the training process of the depth estimation model, a first loss function can be determined based on the difference between the analysis results (depth estimation map) during the training process and the depth mark information; a second loss function can be determined based on the position of the target object in different training views (views in the training data, such as the first view and the second view), combined with the depth estimation map and the view pose relationship between different training views; and the depth estimation model is then trained based on the first loss function and the second loss function, which reduces the dependence of the depth estimation model on the annotated depth mark information and can improve the analysis accuracy of the depth estimation model. After the depth estimation model is trained, it can be applied to the scene of analyzing the relevant data of the live broadcast, such as live broadcast of sports events, online live broadcast, etc. Some wonderful pictures that need to be replayed can be intercepted from the live broadcast data as live broadcast input data, and then the live broadcast input data is input into the depth estimation model. The depth estimation model analyzes the first image features of the first live broadcast view and the second image features of the second live broadcast view, and then determines the second image features mapped to the reference image features of the first live broadcast view based on the posture relationship of the live broadcast view, and adjusts the first image features based on the reference image features, thereby obtaining the depth information map of the first live broadcast view. The depth information map can be understood as the depth estimation map of the depth estimation model. Afterwards, according to the depth information of each object in the depth estimation map, special effects (such as slow motion, magnification and other special effects processing) can be added to the first live broadcast view, a three-dimensional scene model can be constructed, scene recognition, image synthesis and other processing can be performed.

[0140] Based on the above embodiment, this embodiment further provides a data processing device, referring to Fig.11 , specifically it can include the following modules:

[0141] The video data acquisition module 1102 is used to acquire video input data, which includes: first video data to be analyzed, at least one second video data, and a video posture relationship representing the positional relationship between the first video data and the second video data, and the first video data and the second video data contain a target object.

[0142] The video depth map acquisition module 1104 is used to input the video input data into the trained depth estimation model to determine the depth information map of the first video data. The depth estimation model is adjusted based on the first loss function and the second loss function. The first loss function is determined based on the depth estimation map and depth marking information during the training process, and the second loss function is determined based on the position of the key points of the target object in different training views.

[0143] The video special effect processing module 1106 is used to perform special effect processing on the first video data according to the depth information map to form processed video data.

[0144] In summary, in the embodiment of the present application, during the training process of the depth estimation model, the first loss function can be determined based on the difference between the analysis result (depth estimation map) during the training process and the depth labeling information; the second loss function can be determined based on the position of the target object in different training views, combined with the depth estimation map and the view pose relationship between different training views; then the depth estimation model is trained based on the first loss function and the second loss function, which reduces the dependence of the depth estimation model on the annotated depth labeling information and can improve the analysis accuracy of the depth estimation model. After the depth estimation model is trained, it can be applied in the scene of analyzing the video input data, such as the scene of analyzing the video data such as the traffic monitoring video and the security monitoring video, the video input data can be input into the depth estimation model, the depth estimation model analyzes the first image feature of the first video data and the second image feature of the second video data, and then determines the second image feature to be mapped to the reference image feature of the first video data based on the video pose relationship, and adjusts the first image feature based on the reference image feature, thereby obtaining the depth information map of the first video data, and the depth information map can be understood as the depth estimation map of the depth estimation model. Afterwards, special effects may be added to the first video data, a three-dimensional scene model may be constructed, scene recognition may be performed, and image synthesis may be performed, etc., based on the depth information of each object in the depth estimation image.

[0145] Based on the above embodiment, this embodiment further provides a data processing device, referring to Fig.12 , specifically it can include the following modules:

[0146] The interactive page providing module 1202 is used to provide a data interactive page, and the data interactive page includes a data upload control.

[0147] The input data determination module 1204 is used to obtain input data and upload the input data based on the triggering of the data upload control, wherein the input data includes: a first input view to be analyzed, at least one second input view, and an input view posture relationship representing the positional relationship between the first input view and the second input view, wherein the first input view and the second input view contain an object to be analyzed.

[0148] The depth information map output module 1206 is used to display the depth information map of the first input view on the data interaction page, wherein the depth estimation map is determined by inputting the input data into a trained depth estimation model, and the depth estimation model is adjusted according to a first loss function and a second loss function. The first loss function is determined according to the depth estimation map and depth marking information during the training process, and the second loss function is determined according to the positions of the key points of the target object in different training views.

[0149] In an embodiment of the present application, a data interaction page can be provided in the terminal, and the user can upload input data by triggering the data upload control. The processing device can use the trained depth estimation model to analyze the first image features of the first input view and the second image features of the second input view, and then determine the reference image features of the first input view mapped to the second image features based on the input view posture relationship, and adjust the first image features based on the reference image features, thereby obtaining a depth information map of the first input view. The depth information map can be understood as a depth estimation map of the depth estimation model during the training process. After determining the depth information map, the processing device can send the depth information map to the terminal, and the terminal outputs the depth information map through the data interaction page, such as displaying the depth information map in the data interaction page, or providing a download link on the data interaction page so that the user can download the depth information map.

[0150] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0151] The present application embodiment provides one or more machine-readable media, on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more methods as described in the above embodiments. In the present application embodiment, the electronic device includes a server, a terminal device, and the like.

[0152] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as a server (cluster), a terminal, etc. Fig.13 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically illustrated.

[0153] For one embodiment, Fig.13 An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0154] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a server, terminal, or other device described in the embodiments of the present application.

[0155] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0156] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0157] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0158] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1306 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0159] For one embodiment, control module 1304 may include one or more input / output controllers to provide an interface to NVM / storage device 1308 and input / output device(s) 1310 .

[0160] For example, NVM / storage 1308 may be used to store data and / or instructions 1314. NVM / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0161] NVM / storage device 1308 may include storage resources that are part of the device on which apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1308 may be accessed via (one or more) input / output devices 1310 over a network.

[0162] (One or more) input / output devices 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, an audio component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0163] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip (SoC).

[0164] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0165] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0166] An embodiment of the present application further provides an electronic device, comprising: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0167] The embodiments of the present application also provide one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0168] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0169] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0170] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0171] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0173] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0174] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0175] The data processing method, the data processing device, the electronic device and the storage medium provided by the present application are introduced in detail above. The principles and implementation methods of the present application are explained in this article by using specific examples. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, the method comprising: Acquire training input data, and input the training input data into a depth estimation model, determine a depth estimation map of a first view, and determine a first loss function based on the depth estimation map, wherein the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, wherein the first view and the second view contain a target object, and the view pose relationship refers to a relationship between a position and a posture of an acquisition device corresponding to the first view and the second view; Acquire first position information of the key point in the first view and second position information of the key point in the second view; Determining mapping position information in which the first position information is mapped to the second view based on the depth estimation map and the view pose relationship; Determining a second loss function according to the second position information and the mapping position information; The depth estimation model is adjusted according to the first loss function and the second loss function.

2. The method according to claim 1, wherein the depth estimation model is used to perform the following steps: Acquire a first image feature of the first view and a second image feature of the second view; Determining reference image features of the first view mapped from the second view according to the view pose relationship and the second image features; The first image feature is adjusted according to the reference image feature to obtain a depth estimation map of the first view.

3. The method according to claim 1, wherein determining the first loss function according to the depth estimation map comprises: A first loss function is determined according to the depth estimation map and the depth tag information of the first view.

4. The method according to claim 1, wherein determining, based on the depth estimation map and the view pose relationship, mapping the first position information to the mapping position information in the second view comprises: Determining first depth information corresponding to the first position information according to the depth estimation map; Determining second depth information of a key point in a second view according to the view pose relationship and the first depth information; Mapping position information of the key point of the second view is determined according to the second depth information.

5. The method according to claim 1, further comprising: Acquire input data, the input data comprising: a first input view to be analyzed, at least one second input view, and an input view pose relationship representing a positional relationship between the first input view and the second input view, wherein the first input view and the second input view contain a target object; The input data is fed into the trained depth estimation model to determine a depth information map of the first view.

6. The method according to claim 5, further comprising: According to the depth information map, image processing is performed on the first view, where the image processing includes at least one of special effect processing, image synthesis processing, image scene recognition and three-dimensional modeling processing.

7. The method according to claim 1, wherein the first loss function comprises a supervised loss function, and the second loss function comprises an unsupervised loss function.

8. A data processing method, comprising: Acquire input data, the input data comprising: a first input view to be analyzed, at least one second input view, and an input view pose relationship representing a positional relationship between the first input view and the second input view, the first input view and the second input view containing an object to be analyzed; Input data is input into a trained depth estimation model to determine a depth information map of a first input view, the depth estimation model is adjusted based on a first loss function and a second loss function, the first loss function is determined based on a depth estimation map and depth marking information of the first view determined by inputting the training input data into the depth estimation model, the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, the first view and the second view contain a target object, the view pose relationship refers to a relationship between a position and a posture of an acquisition device corresponding to the first view and the second view, the second loss function is determined based on the second position information and a mapping position information of the first position information mapped to the second view, the mapping position information is based on the depth estimation map and the view pose relationship, the first position information is the first position information of a key point in the first view, and the second position information is the second position information of a key point in the second view.

9. A data processing method, comprising: Acquire live broadcast data, and intercept live broadcast input data from the live broadcast data, wherein the live broadcast input data includes: a first live broadcast view to be analyzed, at least one second live broadcast view, and a live broadcast view posture relationship representing a positional relationship between the first live broadcast view and the second live broadcast view, wherein the first live broadcast view and the second live broadcast view include a target object; Input data is input into a trained depth estimation model to determine a depth information map of a first live view, the depth estimation model is adjusted according to a first loss function and a second loss function, the first loss function is determined according to a depth estimation map and depth marking information of a first view determined by inputting training input data into the depth estimation model, the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, the first view and the second view contain a target object, the view pose relationship refers to a relationship between a position and a pose of a collection device corresponding to the first view and the second view, the second loss function maps the second position information and the first position information to a mapping position information in the second view, the mapping position information is determined based on the depth estimation map and the view pose relationship, the first position information is the first position information of a key point in the first view, and the second position information is the second position information of a key point in the second view; According to the depth information map, special effects processing is performed on the first live view to form live playback data.

10. A data processing method, comprising: Acquire video input data, the video input data comprising: first video data to be analyzed, at least one second video data, and a video posture relationship representing a positional relationship between the first video data and the second video data, the first video data and the second video data containing a target object; Input the video input data into a trained depth estimation model to determine a depth information map of the first video data, the depth estimation model is adjusted according to a first loss function and a second loss function, the first loss function is determined according to a depth estimation map and depth marking information of a first view determined by inputting the training input data into the depth estimation model, the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, the first view and the second view contain a target object, the view pose relationship refers to a relationship between a position and a pose of a collection device corresponding to the first view and the second view, the second loss function maps the second position information and the first position information to a mapping position information in the second view, the mapping position information is determined based on the depth estimation map and the view pose relationship, the first position information is the first position information of a key point in the first view, and the second position information is the second position information of a key point in the second view; The first video data is processed with special effects according to the depth information map to form processed video data.

11. A data processing method, comprising: Providing a data interaction page, wherein the data interaction page includes a data upload control; According to triggering the data upload control, input data is acquired and uploaded, wherein the input data includes: a first input view to be analyzed, at least one second input view, and an input view posture relationship representing a positional relationship between the first input view and the second input view, wherein the first input view and the second input view include an object to be analyzed; On the data interaction page, a depth information map of the first input view is displayed, the depth information map is determined by inputting the input data into a trained depth estimation model, the depth estimation model is adjusted according to a first loss function and a second loss function, the first loss function is determined according to the depth estimation map and depth marking information of the first view determined by inputting the training input data into the depth estimation model, the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, the first view and the second view contain a target object, the view pose relationship refers to a relationship between the position and pose of a collection device corresponding to the first view and the second view, the second loss function is determined according to the second position information and the mapping position information of the first position information mapped to the second view, the mapping position information is based on the depth estimation map and the view pose relationship, the first position information is the first position information of the key point in the first view, and the second position information is the second position information of the key point in the second view.

12. A data processing device, comprising: A first function acquisition module is used to acquire training input data and input the training input data into a depth estimation model to determine a depth estimation map of a first view, so as to determine a first loss function according to the depth estimation map, wherein the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, wherein the first view and the second view include a target object, and the view pose relationship refers to a relationship between a position and a posture of a collection device corresponding to the first view and the second view; A second function acquisition module, used to acquire first position information of a key point in the first view and second position information of a key point in the second view; Determining mapping position information in which the first position information is mapped to the second view based on the depth estimation map and the view pose relationship; Determining a second loss function according to the second position information and the mapping position information; An estimation model adjustment module is used to adjust the depth estimation model according to the first loss function and the second loss function.

13. A data processing device, comprising: An input data acquisition module is used to acquire input data, wherein the input data includes: a first input view to be analyzed, at least one second input view, and an input view posture relationship representing a positional relationship between the first input view and the second input view, wherein the first input view and the second input view contain an object to be analyzed; A depth information map acquisition module is used to input input data into a trained depth estimation model to determine a depth information map of a first input view, wherein the depth estimation model is adjusted based on a first loss function and a second loss function, wherein the first loss function is determined based on a depth estimation map and depth marking information of the first view determined by inputting training input data into the depth estimation model, wherein the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, wherein the first view and the second view contain a target object, wherein the view pose relationship refers to a relationship between a position and a posture of an acquisition device corresponding to the first view and the second view, wherein the second loss function is determined based on second position information and mapping position information of the first position information mapped to the second view, wherein the mapping position information is based on the depth estimation map and the view pose relationship, wherein the first position information is first position information of a key point in the first view, and the second position information is second position information of a key point in the second view.

14. A data processing device, comprising: A live broadcast data acquisition module is used to acquire live broadcast data and intercept live broadcast input data from the live broadcast data, wherein the live broadcast input data includes: a first live broadcast view to be analyzed, at least one second live broadcast view, and a live broadcast view posture relationship representing a positional relationship between the first live broadcast view and the second live broadcast view, wherein the first live broadcast view and the second live broadcast view contain a target object; A live depth map acquisition module, used to input input data into a trained depth estimation model to determine a depth information map of a first live view, the depth estimation model is adjusted according to a first loss function and a second loss function, the first loss function is determined according to a depth estimation map and depth marking information of a first view determined by inputting training input data into the depth estimation model, the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, the first view and the second view contain a target object, the view pose relationship refers to a relationship between a position and a pose of a collection device corresponding to the first view and the second view, the second loss function is determined according to second position information and mapping position information of the first position information mapped to the second view, the mapping position information is based on the depth estimation map and the view pose relationship, the first position information is first position information of a key point in the first view, and the second position information is second position information of a key point in the second view; The live broadcast special effect processing module is used to perform special effect processing on the first live broadcast view according to the depth information map to form live broadcast playback data.

15. A data processing device, comprising: A video data acquisition module, used to acquire video input data, wherein the video input data includes: first video data to be analyzed, at least one second video data, and a video posture relationship representing a positional relationship between the first video data and the second video data, wherein the first video data and the second video data contain a target object; A video depth map acquisition module, used for inputting video input data into a trained depth estimation model to determine a depth information map of first video data, wherein the depth estimation model is adjusted according to a first loss function and a second loss function, wherein the first loss function is determined according to a depth estimation map and depth marking information of a first view determined by inputting training input data into the depth estimation model, wherein the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, wherein the first view and the second view contain a target object, wherein the view pose relationship refers to a relationship between a position and a pose of a collection device corresponding to the first view and the second view, wherein the second loss function is determined according to second position information and mapping position information of the first position information mapped to the second view, wherein the mapping position information is based on the depth estimation map and the view pose relationship, wherein the first position information is first position information of a key point in the first view, and the second position information is second position information of a key point in the second view; The video special effect processing module is used to perform special effect processing on the first video data according to the depth information map to form processed video data.

16. A data processing device, comprising: An interactive page providing module, used to provide a data interactive page, wherein the data interactive page includes a data upload control; An input data determination module is used to obtain input data according to the triggering of the data upload control, and upload the input data, wherein the input data includes: a first input view to be analyzed, at least one second input view, and an input view posture relationship representing a positional relationship between the first input view and the second input view, wherein the first input view and the second input view contain an object to be analyzed; A depth information map output module is used to display a depth information map of a first input view on a data interaction page, wherein the depth information map is determined by inputting input data into a trained depth estimation model, wherein the depth estimation model is adjusted according to a first loss function and a second loss function, wherein the first loss function is determined according to a depth estimation map and depth marking information of a first view determined by inputting training input data into the depth estimation model, wherein the training input data includes a first view to be analyzed, at least one second view, and a view pose relationship between the first view and the second view, wherein the first view and the second view contain a target object, wherein the view pose relationship refers to a relationship between a position and a pose of a collection device corresponding to the first view and the second view, wherein the second loss function is determined according to second position information and mapping position information of the first position information mapped to the second view, wherein the mapping position information is based on the depth estimation map and the view pose relationship, wherein the first position information is first position information of a key point in the first view, and the second position information is second position information of a key point in the second view.

17. An electronic device comprising: processor; and A memory having executable codes stored thereon, which, when executed, cause the processor to perform the method according to one or more of claims 1-11.

18. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the method of one or more of claims 1-11.

Images