Picture processing method and device, equipment and storage medium

Abstract

The invention provides a picture processing method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. In some embodiments of the disclosure, an original picture is input into an encoder for feature encoding, and key features of the original picture are obtained; the key features of the original picture and the key features of the historical picture are input into a feature memory network for feature fusion, and feature fusion information is obtained, the feature fusion information comprises Gaussian parameters, point cloud information and depth estimation information, and the Gaussian parameters are predicted through the network; high-quality Gaussian parameters, point cloud information and depth estimation information are input into an online back propagation network, the number of back propagation times can be reduced, the picture processing efficiency is improved, the updated Gaussian parameters with higher precision are obtained by adopting a back propagation method, the method can be suitable for various task models, and the image processing efficiency is improved. And the compatibility of the model to various task scenes is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] Task Awareness means that when executing a task, the system can dynamically adjust its operation strategy and parameters according to the current task requirements and environmental changes to adapt to different task scenarios and goals. This ability is particularly important in the fields of autonomous driving, robot control, and target detection.

[0003] In scenarios such as robots, autonomous driving, and drones, the perception tasks include real-time new perspective synthesis, monocular estimation, mapping, positioning, object classification, recognition, segmentation, tracking, and feature point extraction.

[0004] Currently, the multi-task model can only complete limited perception tasks, has low scene compatibility, and low image processing efficiency. Summary of the invention

[0005] The present disclosure provides an image processing method, apparatus, device and storage medium to at least solve the problems of low scene compatibility and low image processing efficiency of existing multi-task models.

[0006] The technical solution of the present disclosure is as follows: The present disclosure provides a method for processing an image, including: Input the original image into the encoder for feature encoding to obtain the key features of the original image; Inputting the key features of the original image and the key features of the historical image into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; Input the Gaussian parameters, point cloud information and depth estimation information into the online back propagation network for reverse iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information; The updated Gaussian parameters are input into the predictor corresponding to each task type to obtain the prediction results, wherein the accuracy of the updated Gaussian parameters is higher than that of the Gaussian parameters.

[0007] Optionally, the feature fusion information also includes: object recognition information, object segmentation information, object tracking information, feature point extraction information; the task types include: segmentation, detection, recognition, monocular estimation, mapping, positioning, new perspective synthesis and feature point extraction.

[0008] Optionally, the encoder includes: a tile embedding layer, a position encoding layer and a plurality of encoder layers, wherein the plurality of encoder layers are serially connected; the inputting the original image into the encoder for feature encoding to obtain key features of the original image includes: Inside the encoder, inputting the original image into the tile embedding layer to obtain a tile embedding; Inputting the tile embedding vector into the position encoding layer to obtain a position-encoded tile embedding; The position-encoded image blocks are embedded into the encoder layer to obtain the key features of the original image.

[0009] Optionally, each of the encoder layers includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer, and a second residual connection and layer normalization layer; the embedding of the position-encoded image block into the encoder layer to obtain the key features of the original image includes: For a target encoder layer, embedding the position-encoded image block into a multi-head self-attention mechanism layer input into the target encoder layer to obtain a self-attention output feature; wherein the target encoder layer is any one of the multiple encoder layers; Embedding the self-attention output feature and the position-encoded tile into the first residual connection and layer normalization layer of the target encoder layer to obtain a first normalized feature; Inputting the first normalized feature into the feedforward neural network layer of the target encoder layer to obtain a feedforward network output feature; The feedforward network output feature and the first normalized feature are input into the second residual connection and layer normalization layer of the target encoder layer to obtain a second normalized feature.

[0010] Optionally, the feature memory network includes: a plurality of basic block modules and head modules connected in series, and the inputting of the key features of the original image and the key features of the historical image into the feature memory network for feature fusion to obtain feature fusion information includes: Inputting the original image key features and the historical image key features into a plurality of the basic block modules to obtain initial fusion features; The initial fusion features are input into the head module to obtain the feature fusion information.

[0011] Optionally, the step of inputting the Gaussian parameters, the point cloud information, and the depth estimation information into an online back propagation network for reverse iteration to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information includes: Inputting the Gaussian parameters, the point cloud information, and the depth estimation information into an online back propagation network, rendering the original image, and obtaining a rendered image corresponding to the original image; Determining a loss function according to the rendered image and the original image; Back propagation is performed according to the loss function to update the Gaussian parameters, the point cloud information and the depth estimation information to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information.

[0012] The present disclosure also provides a picture processing device, including: The encoding module is used to input the original image into the encoder for feature encoding to obtain the key features of the original image; A fusion module, used for inputting the key features of the original image and the key features of the historical image into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; A rendering module, used for inputting Gaussian parameters, the point cloud information and the depth estimation information into an online back propagation network for iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information; A prediction module is used to input the updated Gaussian parameters into a predictor corresponding to each task type to obtain a prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the Gaussian parameters.

[0013] The present disclosure also provides an electronic device, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement each step in the above method.

[0014] The embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the above method is implemented.

[0015] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements each step of the above method when executed by a processor.

[0016] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: In some embodiments of the present disclosure, the original image is input into an encoder for feature encoding to obtain key features of the original image; the key features of the original image and the key features of the historical image are input into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information. The present disclosure predicts Gaussian parameters through a network; high-quality Gaussian parameters, point cloud information and depth estimation information are input into an online back propagation network, which can reduce the number of back propagation times and improve image processing efficiency, and the back propagation method is used to obtain higher-precision updated Gaussian parameters, which can be applicable to a variety of task models and improve the compatibility of the model for various task scenarios.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0019] Figure 1 A schematic diagram of a process of image processing provided by an exemplary embodiment of the present disclosure; Figure 2 A schematic diagram of the structure of an image processing system provided by an exemplary embodiment of the present disclosure; Figure 3 A schematic diagram of the structure of an online back propagation network provided for an exemplary embodiment of the present disclosure; Figure 4 A schematic diagram of the structure of a picture processing device provided by an exemplary embodiment of the present disclosure; Figure 5 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0022] It should be noted that the user information involved in this disclosure includes but is not limited to: user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure comply with the relevant laws and regulations and do not violate public order and good morals.

[0023] The following is a description of the existing perception task scheme.

[0024] 1. Mapping and positioning feature point extraction scheme. The input is multiple pictures. First, the feature points of each picture are obtained. Then, through feature point matching, the depth of the feature points and the posture of each picture are calculated to complete the mapping and positioning. This scheme cannot be applied to many scenarios and has low universality. Second, the monocular estimation solution. The input is a single image. The network structure includes an encoder and a decoder. The output is a depth map. The encoder is a feature extraction module and the decoder is a decoding module, which is used to predict the depth based on the features. This solution cannot be applied to a variety of scenarios and has low universality.

[0025] 3. The scheme for synthesizing new perspectives. The input is multiple images and an initial sparse point cloud. Then the Gaussian parameters are initialized to a preset value. For example, the opcity parameter is initialized to 0.1. Then the parameters are updated through back propagation. Each back propagation will update the parameters once to make them "a little more correct". It usually takes 30,000 back iterations to obtain sufficiently correct parameters. Then, based on these parameters, the Gaussian rendering engine can be used to render images with new perspectives. The image rendering time is long, usually 10 minutes to 1 hour.

[0026] In response to the above technical problems, in some embodiments of the present disclosure, the original image is input into an encoder for feature encoding to obtain key features of the original image; the key features of the original image and the key features of the historical image are input into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information. The present disclosure predicts Gaussian parameters through a network; high-quality Gaussian parameters, point cloud information and depth estimation information are input into an online back propagation network, which can reduce the number of back propagation times and improve image processing efficiency, and the back propagation method is used to obtain higher-precision updated Gaussian parameters, which can be applicable to a variety of task models and improve the compatibility of the model for various task scenarios.

[0027] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0028] Figure 1 The following is a flow chart of a method for image processing provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, the method includes: S101: Input the original image into the encoder for feature encoding to obtain key features of the original image; S102: inputting the key features of the original image and the key features of the historical image into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; S103: inputting Gaussian parameters, point cloud information and depth estimation information into an online back propagation network for reverse iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information; S104: Inputting the updated Gaussian parameters into the predictor corresponding to each task type to obtain a prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the Gaussian parameters.

[0029] In this embodiment, the execution subject of the above method may be a terminal device or a server.

[0030] Among them, the terminal device includes but is not limited to a mobile station (MS), a mobile terminal, a mobile phone, a handset and a portable equipment, etc. The terminal device can communicate with one or more core networks via a radio access network (RAN). For example, the terminal device can be a mobile phone (or a "cellular" phone), a computer with a wireless communication function, etc. The terminal device can also be a computer with a wireless transceiver function, a virtual reality (VR) terminal device, an AR terminal device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc., and the operating system installed on the terminal device includes but is not limited to: IOS, Android, windows, linux, Mac OS and other operating systems. In different networks, the terminal may be called different names, such as user equipment, mobile station, subscriber unit, station, cellular phone, personal digital assistant, wireless modem, wireless communication device, handheld device, laptop computer, cordless phone, wireless local loop station, television, etc. For the convenience of description, it is referred to as terminal equipment in this embodiment.

[0031] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly includes a processor, a hard disk, a memory, a system bus, etc., and a general computer architecture type.

[0032] In this embodiment, in some embodiments of the present disclosure, the original image is input into the encoder for feature encoding to obtain the key features of the original image; the key features of the original image and the key features of the historical image are input into the feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information, and the present disclosure predicts Gaussian parameters through the network; high-quality Gaussian parameters, point cloud information and depth estimation information are input into the online back propagation network, which can reduce the number of back propagation times and improve image processing efficiency, and the back propagation method is used to obtain higher-precision updated Gaussian parameters, which can be applicable to a variety of task models and improve the compatibility of the model for various task scenarios.

[0033] In this embodiment, the original image is input into the encoder for feature encoding to obtain the key features of the original image; the key features of the original image and the key features of the historical images are input into the feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information, and the present invention predicts Gaussian parameters through the network; high-quality Gaussian parameters, point cloud information and depth estimation information are input into the online back propagation network, and the original image is rendered to obtain a rendered image corresponding to the original image, which can reduce the number of back propagation times and improve image processing efficiency, and use the back propagation method to update parameters to improve the compatibility of the model for various scenarios.

[0034] It should be noted that Gaussian parameters include: mean vector, covariance matrix, weight, amplitude and scale parameter.

[0035] Among them, the mean vector (Mean Vector), symbolized as μ, defines the center position of the Gaussian distribution. In three-dimensional space, the mean vector is usually expressed as μ = (μ_x, μ_y, μ_z), where μ_x, μ_y and μ_z are the center coordinates of the Gaussian distribution in the x, y and z directions respectively. The covariance matrix (Covariance Matrix), symbolized as Σ, defines the shape and direction of the Gaussian distribution. In three-dimensional space, the covariance matrix is ​​a 3x3 symmetric positive definite matrix, expressed as: Σ = [σ_xxσ_xyσ_xz][σ_yxσ_yyσ_yz][σ_zxσ_zyσ_zz] Among them, σ_xx, σ_yy and σ_zz are the variances in the x, y and z directions, while σ_xy, σ_xz, etc. are the covariances in different directions. The covariance matrix determines the spread and rotation of the Gaussian distribution.

[0036] Among them, the weight (Weight), symbolized as w, determines the influence or importance of the Gaussian distribution. During the rendering process, different Gaussian distributions may have different weights to indicate their contribution to the final image.

[0037] Amplitude, denoted by A, determines the height or strength of the Gaussian distribution. It is a scalar value that controls the maximum value of the Gaussian distribution at its center.

[0038] Among them, the scale parameters are denoted as s_x, s_y, s_z; these parameters control the extent of the Gaussian distribution in each direction and can usually be derived from the covariance matrix.

[0039] In practical applications, these parameters jointly determine the shape, position, and influence of the Gaussian distribution, thus forming a complex distribution in three-dimensional space for representing and processing three-dimensional data points.

[0040] In this embodiment, the encoder includes: a tile embedding layer, a position encoding layer, and multiple encoder layers, and the multiple encoder layers are serially connected.

[0041] In some embodiments of the present disclosure, the original image is input into the encoder for feature encoding to obtain the key features of the original image. Inside the encoder, the original image is input into the tile embedding layer to obtain the tile embedding; the tile embedding vector is input into the position encoding layer to obtain the position-encoded tile embedding; the position-encoded tile embedding is input into the encoder layer to obtain the key features of the original image. It should be noted that the encoder in this embodiment is used to extract features from the original image; the present disclosure does not limit the type of encoder, which can be adjusted according to actual conditions.

[0042] For example, the encoder is a vit encoder; the original image frame t is input into the tile embedding layer, the image is divided into tiles of fixed size, each tile is flattened into a one-dimensional vector, and the flattened tiles are mapped to the embedding space by linear projection to obtain the tile embedding; wherein, the shape of the original image is H×W×C; the shape of the tile embedding is N×D, where N is the number of tiles and D is the embedding dimension. Frame t represents the image acquired at time t, and frame t-1 represents the image acquired at time t-1, that is, the image acquired before time t. The tile embedding vector is input into the position encoding layer, and a learnable position encoding is added to each tile embedding to retain the position information, and the position-encoded tile embedding is obtained, wherein the shape of the position-encoded tile embedding is N×D. The position-encoded tile embedding is input into the encoder layer, and multiple encoder layers are stacked sequentially to obtain the key features of the original image, and the shape of the key features of the original image is N×D.

[0043] Each encoder layer consists of: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer, and a second residual connection and layer normalization layer.

[0044] In some embodiments of the present disclosure, the position-encoded tiles are embedded in the input encoder layer to obtain the key features of the original image. One achievable method is that, for the target encoder layer, the position-encoded tiles are embedded in the multi-head self-attention mechanism layer of the input target encoder layer to obtain the self-attention output features; wherein the target encoder layer is any one of the multiple encoder layers; the self-attention output features and the position-encoded tiles are embedded in the first residual connection and layer normalization layer of the input target encoder layer to obtain the first normalized features; the first normalized features are input into the feedforward neural network layer of the target encoder layer to obtain the feedforward network output features; the feedforward network output features and the first normalized features are input into the second residual connection and layer normalization layer of the target encoder layer to obtain the second normalized features.

[0045] For example, embed the position-encoded tile into the input multi-head self-attention mechanism layer to generate query (Q), key (K), and value (V) matrices; calculate the attention score and weighted sum; multiple attention heads are calculated in parallel, and the results are concatenated and linearly transformed to obtain the self-attention output feature; the shape of the self-attention output feature is N×D. The self-attention output feature and the position-encoded tile are embedded in the first residual connection and layer normalization layer of the input target encoder layer, and the self-attention output feature and the position-encoded tile embedding are added, and the layer is normalized to obtain the first normalized feature; the shape of the first normalized feature is N×D. The first normalized feature is input into the feedforward neural network layer of the target encoder layer, in the two-layer fully connected network, and the activation function is used in the middle to obtain the feedforward network output feature; the shape of the feedforward network output feature is N×D. The feedforward network output feature and the first normalized feature are input into the second residual connection and layer normalization layer of the target encoder layer, and the feedforward network output feature and the first normalized feature are added, and the layer normalization is applied to obtain the second normalized feature. When the target encoder layer is the last encoder layer in the encoder layer, the second normalized feature is the key feature of the original image.

[0046] In this embodiment, the feature memory network includes: a plurality of serially connected basic block modules and a head module. Each basic block module is composed of a traditional self-attention and cross attention structure. The head module is composed of multiple layers of 3*3 convolutions. The input of the convolution kernel of the first layer is n and the output is 4*n. The input of the convolution kernel of the first layer is 4*n and the output is 7+3+m+1+20. 7 represents the Gaussian parameter, 3 represents the point cloud coordinates, and the depth estimation can be directly calculated from the point cloud coordinates. m represents the number of categories of object segmentation, 1 represents the id of object tracking, and 20 represents the dimension of the feature point.

[0047] In some embodiments of the present disclosure, the key features of the original image and the key features of the historical image are input into the feature memory network for feature fusion to obtain feature fusion information. One feasible way is to input the key features of the original image and the key features of the historical image into multiple basic block modules to obtain initial fusion features; and input the initial fusion features into the head module to obtain feature fusion information. It should be noted that the features of frame t and feature memory t-1 are input into the FM module together; when t is 1, frame 1 is regarded as feature memory t-1. Among them, feature memory t-1 represents the collection of key features of historical images. Through the feature memory network, feature memory t can be continuously updated, and feature t can be helped to fuse historical features to improve the representation ability of features.

[0048] It should be noted that the feature fusion information includes but is not limited to: Gaussian parameters, point cloud information, depth estimation information, object recognition information, object segmentation information, object tracking information and feature point extraction information.

[0049] Figure 2 FIG. 1 is a schematic diagram of a structure of an image processing system provided by an exemplary embodiment of the present disclosure. Figure 2 As shown, the original image frame t is input into the encoder to obtain the key feature feature t of the original image; the key feature feature t of the original image and the key features feature t-1...feature tn of the historical images are input into the feature memory network for feature fusion to obtain Gaussian parameters, point cloud information, depth estimation information, object recognition information, object segmentation information, object tracking information and feature point extraction information.

[0050] In some embodiments of the present disclosure, Gaussian parameters, point cloud information, and depth estimation information are input into an online back propagation network for reverse iteration to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information. One achievable method is to input Gaussian parameters, point cloud information, and depth estimation information into an online back propagation network, render the original image, and obtain a rendered image corresponding to the original image; determine a loss function based on the rendered image and the original image; perform back propagation based on the loss function, update the Gaussian parameters, point cloud information, and depth estimation information, and obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information.

[0051] For example, Figure 3 FIG. 1 is a schematic diagram of the structure of an online back propagation network provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the Gaussian parameters, point cloud information and depth estimation information are input into the online back propagation network, and a rendered image corresponding to the original image is rendered according to the predicted Gaussian parameters. The loss function is determined according to the rendered image and the original image, wherein the loss function may be an L2 loss rendering loss function, and the L2 loss is back propagated 10 times for each Gaussian parameter to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information.

[0052] In some implementations of the present disclosure, the updated Gaussian parameters are input into the predictor corresponding to each task type to obtain a prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the Gaussian parameters. Among them, the task types include: segmentation, detection, recognition, monocular estimation, mapping, positioning, new perspective synthesis and feature point extraction. The present disclosure adopts the back propagation method to obtain the updated Gaussian parameters with higher accuracy, which can be applied to a variety of task models and improve the compatibility of the model for various task scenarios.

[0053] Figure 4 FIG. 4 is a schematic diagram of a structure of an image processing device 40 provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the image processing device 40 includes: an encoding module 41, a fusion module 42, a rendering module 43 and a prediction module 44.

[0054] The encoding module 41 is used to input the original image into the encoder for feature encoding to obtain the key features of the original image; A fusion module 42 is used to input the key features of the original image and the key features of the historical image into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; A rendering module 43, used for inputting Gaussian parameters, point cloud information and depth estimation information into an online back propagation network for iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information; The prediction module 44 is used to input the updated Gaussian parameters into the predictor corresponding to each task type to obtain a prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the Gaussian parameters.

[0055] Optionally, the feature fusion information also includes: object recognition information, object segmentation information, object tracking information and feature point extraction information; task types include: segmentation, detection, recognition, monocular estimation, mapping, positioning, new perspective synthesis and feature point extraction.

[0056] Optionally, the encoder includes: a tile embedding layer, a position encoding layer and multiple encoder layers, and the multiple encoder layers are serially connected; when the encoding module 41 inputs the original image into the encoder for feature encoding to obtain the key features of the original image, it is used to: Inside the encoder, the original image is input into the tile embedding layer to obtain the tile embedding; Input the tile embedding vector into the position encoding layer to obtain the position-encoded tile embedding; The position-encoded tiles are embedded into the input encoder layer to obtain the key features of the original image.

[0057] Optionally, each encoder layer includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer, and a second residual connection and layer normalization layer; when the encoding module 41 embeds the position-encoded image block into the input encoder layer to obtain the key features of the original image, it is used to: For the target encoder layer, embed the position-encoded tiles into the multi-head self-attention mechanism layer of the input target encoder layer to obtain the self-attention output features; wherein the target encoder layer is any one of the multiple encoder layers; Embed the self-attention output features and the position-encoded tiles into the first residual connection and layer normalization layer of the input target encoder layer to obtain the first normalized features; Input the first normalized feature into the feedforward neural network layer of the target encoder layer to obtain the feedforward network output feature; The feedforward network output features and the first normalized features are input into the second residual connection and layer normalization layer of the target encoder layer to obtain the second normalized features.

[0058] Optionally, the feature memory network includes: a plurality of basic block modules and head modules connected in series, and the fusion module 42 inputs the key features of the original image and the key features of the historical image into the feature memory network for feature fusion to obtain feature fusion information, and is used to: Input the key features of the original image and the key features of the historical image into multiple basic block modules to obtain the initial fusion features; The initial fusion features are input into the head module to obtain feature fusion information.

[0059] Optionally, when the rendering module 43 inputs the Gaussian parameters, the point cloud information and the depth estimation information into the online back propagation network for reverse iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information, it is used to: Input Gaussian parameters, point cloud information, and depth estimation information into the online back propagation network, render the original image, and obtain a rendered image corresponding to the original image; Determine the loss function based on the rendered image and the original image; Back propagation is performed according to the loss function, and the Gaussian parameters, point cloud information, and depth estimation information are reversely iterated to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information.

[0060] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0061] Figure 5 FIG. 1 is a schematic diagram of a structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the electronic device includes: a memory 51 and a processor 52. In addition, the electronic device also includes a power supply component 53 and a communication component 54.

[0062] The memory 51 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.

[0063] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0064] The communication component 54 is used for data transmission with other devices.

[0065] The processor 52 can execute computer instructions stored in the memory 51 to: input the original image into the encoder for feature encoding to obtain the key features of the original image; input the key features of the original image and the key features of the historical image into the feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; perform reverse iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information; input the updated Gaussian parameters into the predictor corresponding to each task type to obtain a prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the Gaussian parameters.

[0066] Optionally, the feature fusion information also includes: object recognition information, object segmentation information, object tracking information and feature point extraction information; task types include: segmentation, detection, recognition, monocular estimation, mapping, positioning, new perspective synthesis and feature point extraction.

[0067] Optionally, the encoder includes: a tile embedding layer, a position encoding layer and a plurality of encoder layers, and the plurality of encoder layers are serially connected; when the processor 52 inputs the original image into the encoder for feature encoding to obtain the key features of the original image, it is used to: Inside the encoder, the original image is input into the tile embedding layer to obtain the tile embedding; Input the tile embedding vector into the position encoding layer to obtain the position-encoded tile embedding; The position-encoded tiles are embedded into the input encoder layer to obtain the key features of the original image.

[0068] Optionally, each encoder layer includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer, and a second residual connection and layer normalization layer; when the processor 52 embeds the position-encoded image block into the input encoder layer to obtain the key features of the original image, it is used to: For the target encoder layer, embed the position-encoded tiles into the multi-head self-attention mechanism layer of the input target encoder layer to obtain the self-attention output features; wherein the target encoder layer is any one of the multiple encoder layers; Embed the self-attention output features and the position-encoded tiles into the first residual connection and layer normalization layer of the input target encoder layer to obtain the first normalized features; Input the first normalized feature into the feedforward neural network layer of the target encoder layer to obtain the feedforward network output feature; The feedforward network output features and the first normalized features are input into the second residual connection and layer normalization layer of the target encoder layer to obtain the second normalized features.

[0069] Optionally, the feature memory network includes: a plurality of basic block modules and head modules connected in series, and the processor 52 inputs the key features of the original image and the key features of the historical image into the feature memory network for feature fusion to obtain feature fusion information, and is used to: Input the key features of the original image and the key features of the historical image into multiple basic block modules to obtain the initial fusion features; The initial fusion features are input into the head module to obtain feature fusion information.

[0070] Optionally, when the processor 52 inputs the Gaussian parameters, the point cloud information and the depth estimation information into the online back propagation network for reverse iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information, it is used to: Input Gaussian parameters, point cloud information, and depth estimation information into the online back propagation network, render the original image, and obtain a rendered image corresponding to the original image; Determine the loss function based on the rendered image and the original image; Back propagation is performed according to the loss function, and the Gaussian parameters, point cloud information, and depth estimation information are reversely iterated to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information.

[0071] Accordingly, the present disclosure also provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 1 Each step in the method embodiment.

[0072] Accordingly, the present disclosure also provides a computer program product, the computer program product includes a computer program / instruction, the computer program / instruction is executed by a processor Figure 1 Each step in the method embodiment.

[0073] Above Figure 5 The communication component in the device is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0074] Above Figure 5 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0075] The electronic device also includes a display screen and an audio component.

[0076] The display screen includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0077] The audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0078] In the above-mentioned method, apparatus, device, storage medium and computer program product embodiments of the present invention, the original image is input into the encoder for feature encoding to obtain the key features of the original image; the key features of the original image and the key features of the historical image are input into the feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information. The present invention predicts Gaussian parameters through the network; high-quality Gaussian parameters, point cloud information and depth estimation information are input into the online back propagation network, which can reduce the number of back propagation times and improve the image processing efficiency, and the back propagation method is used to obtain higher-precision updated Gaussian parameters, which can be applicable to a variety of task models and improve the compatibility of the model for various task scenarios.

[0079] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0083] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0084] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0085] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0086] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0087] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing an image, characterized in that: include: Input the original image into the encoder for feature encoding to obtain the key features of the original image; Inputting the key features of the original image and the key features of the historical image into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; Inputting the Gaussian parameters, the point cloud information and the depth estimation information into an online back propagation network for reverse iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information; The updated Gaussian parameters are input into a predictor corresponding to each task type to obtain a prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the Gaussian parameters.

2. The method according to claim 1, characterized in that The feature fusion information also includes: object recognition information, object segmentation information, object tracking information and feature point extraction information; the task types include: segmentation, detection, recognition, monocular estimation, mapping, positioning, new perspective synthesis and feature point extraction.

3. The method according to claim 1, characterized in that The encoder comprises: a tile embedding layer, a position encoding layer and a plurality of encoder layers, wherein the plurality of encoder layers are serially connected; the original image is input into the encoder for feature encoding to obtain key features of the original image, including: Inside the encoder, inputting the original image into the tile embedding layer to obtain a tile embedding; Inputting the tile embedding vector into the position encoding layer to obtain a position-encoded tile embedding; The position-encoded image blocks are embedded into the encoder layer to obtain the key features of the original image.

4. The method according to claim 3, characterized in that Each of the encoder layers includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer, and a second residual connection and layer normalization layer; the embedding of the position-encoded image blocks into the encoder layer to obtain the key features of the original image includes: For a target encoder layer, embedding the position-encoded image block into a multi-head self-attention mechanism layer input into the target encoder layer to obtain a self-attention output feature; wherein the target encoder layer is any one of the multiple encoder layers; Embedding the self-attention output feature and the position-encoded tile into the first residual connection and layer normalization layer of the target encoder layer to obtain a first normalized feature; Inputting the first normalized feature into the feedforward neural network layer of the target encoder layer to obtain a feedforward network output feature; The feedforward network output feature and the first normalized feature are input into the second residual connection and layer normalization layer of the target encoder layer to obtain a second normalized feature.

5. The method according to claim 1, characterized in that: The feature memory network includes: a plurality of serially connected basic block modules and head modules, and the key features of the original image and the key features of the historical image are input into the feature memory network for feature fusion to obtain feature fusion information, including: Inputting the original image key features and the historical image key features into a plurality of the basic block modules to obtain initial fusion features; The initial fusion features are input into the head module to obtain the feature fusion information.

6. The method according to claim 1, characterized in that The step of inputting the Gaussian parameters, the point cloud information and the depth estimation information into an online back propagation network for reverse iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information comprises: Inputting the Gaussian parameters, the point cloud information, and the depth estimation information into an online back propagation network, rendering the original image, and obtaining a rendered image corresponding to the original image; Determining a loss function according to the rendered image and the original image; Back propagation is performed according to the loss function, and the Gaussian parameters, the point cloud information, and the depth estimation information are reversely iterated to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information.

7. A picture processing device, characterized in that: include: The encoding module is used to input the original image into the encoder for feature encoding to obtain the key features of the original image; A fusion module, used for inputting the key features of the original image and the key features of the historical image into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; A rendering module, used for inputting Gaussian parameters, the point cloud information and the depth estimation information into an online back propagation network for iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information; A prediction module is used to input the updated Gaussian parameters into a predictor corresponding to each task type to obtain a prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the Gaussian parameters.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement each step in the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.

Images