Data processing method, device, electronic device and storage medium

Abstract

The disclosed embodiments provide a data processing method, device, electronic device and storage medium. The method includes: obtaining target acquisition viewing angle parameters; inputting the target acquisition viewing angle parameters into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting one or more of a first rendering image associated with the target area under the target acquisition viewing angle parameters for characterizing the global semantic features associated with the target acquisition viewing angle parameters, a second rendering image of the spatial features and a third rendering image of the instance features. The technical solution of the disclosed embodiments can obtain a rendering image including global semantic features, a rendering image including spatial features and a rendering image including instance features. These feature information provides three-dimensional spatial constraint information, improves the integrity of the object feature expression, and thus performs three-dimensional scene understanding tasks based on these rendering images, ensuring execution accuracy and reliability.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology, and in particular, to a data processing method, device, electronic device, and storage medium. Background Art

[0002] Computer vision is an important branch of artificial intelligence. It means that computers can replace human eyes and brains to visually recognize an environment and a stimulus (or object) in the environment. The three-dimensional scene understanding task is an important research direction of computer vision. By modeling objects and their semantics in three-dimensional space, more natural human-computer interaction applications such as virtual reality, autonomous driving and robot navigation can be realized.

[0003] At present, the models used to perform 3D scene understanding tasks mainly process 2D images and identify the information content in the 2D images. The processing results of the models are also information related to 2D features. These processing results have the problem of incomplete semantic expression of objects. Furthermore, when these 2D features are applied to 3D scene understanding tasks, there are technical problems of low execution accuracy and poor reliability. Summary of the invention

[0004] The present disclosure provides a data processing method, device, electronic device and storage medium, so as to obtain a rendered image including global semantic features, a rendered image including spatial features and a rendered image including instance features. These feature information provide three-dimensional spatial constraint information, improve the integrity of object feature expression, so as to perform three-dimensional scene understanding tasks based on these rendered images and ensure execution accuracy and reliability.

[0005] In a first aspect, an embodiment of the present disclosure provides a data processing method, the method comprising:

[0006] Get the target acquisition viewing angle parameters;

[0007] Inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter;

[0008] The multiple rendered images include at least one or more of a first rendered image for representing global semantic features associated with target acquisition viewing angle parameters, a second rendered image for spatial features, and a third rendered image for instance features.

[0009] In a second aspect, an embodiment of the present disclosure further provides a data processing device, the device comprising:

[0010] A viewing angle parameter acquisition module is used to obtain the viewing angle parameters of the target acquisition;

[0011] A rendering image determination module, used for inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendering images associated with the target area under the target acquisition viewing angle parameter;

[0012] The multiple rendered images include at least one or more of a first rendered image for representing global semantic features associated with target acquisition viewing angle parameters, a second rendered image for spatial features, and a third rendered image for instance features.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of the embodiments of the present disclosure.

[0017] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the data processing method as described in any one of the embodiments of the present disclosure.

[0018] The technical solution of the disclosed embodiment is applied to the execution of a three-dimensional scene understanding task, by obtaining a target acquisition viewing angle parameter, inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model for multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter, wherein the plurality of rendered images at least include one or more of a first rendered image for representing a global semantic feature associated with the target acquisition viewing angle parameter, a second rendered image for a spatial feature, and a third rendered image for an instance feature. The technical solution of the disclosed embodiment processes the target acquisition viewing angle parameter through a target Gaussian model, and can obtain a rendered image including a global semantic feature, a rendered image including a spatial feature, and a rendered image including an instance feature. These feature information provide three-dimensional spatial constraint information, improve the integrity of the object feature expression, and thereby execute the three-dimensional scene understanding task based on these rendered images, thereby ensuring execution accuracy and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of multiple rendered images output by a target Gaussian model involved in an embodiment of the present disclosure;

[0022] Figure 3 A flowchart of another data processing method provided by an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a process of processing target acquisition viewing angle parameters by a target Gaussian model according to an embodiment of the present disclosure;

[0024] Figure 5 A flowchart of another data processing method provided by an embodiment of the present disclosure;

[0025] Figure 6 A flowchart of another data processing method provided by an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of an implementation process of determining a global sample semantic feature map corresponding to an image to be processed based on a multi-level context-aware model according to an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of a semantic feature loss value according to an embodiment of the present disclosure;

[0028] Fig. 9 A schematic diagram of spatial Gaussian point loss involved in an embodiment of the present disclosure;

[0029] Fig.10 A schematic diagram of the instantiation loss involved in the embodiments of the present disclosure;

[0030] Fig.11 A schematic diagram of the structure of a data processing device provided by an embodiment of the present disclosure;

[0031] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0035] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0038] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0039] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0040] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0041] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0042] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0043] Before introducing the present technical solution, an example description of the application scenario can be given first. The technical solution can be applied in virtual reality scenarios, autonomous driving scenarios, and robot navigation scenarios. Taking the robot navigation scenario as an example, how the solution provided by the embodiment of the present disclosure is applied in the robot navigation scenario is described. Usually, the robot can be applied to a specific three-dimensional space area, and this area can be used as a target area. The target area includes multiple objects. For example, these objects may include bookcases, tables, multiple chairs, multiple cups, and the like. According to the solution provided by the embodiment of the present disclosure, the images to be processed of the target area under various camera viewing angle parameters can be collected in advance, and the Gaussian model can be trained by these images to be processed to obtain the target Gaussian model. The solution provided by the embodiment of the present disclosure can be integrated into the program of the robot. When the robot is required to perform an object grasping task of grasping an object in the target area, the robot can receive an object grasping instruction. At this time, the perspective parameter of the perspective perception device configured on the robot in the space is the target acquisition perspective parameter; this target acquisition perspective parameter can be input into the target Gaussian model, and the target Gaussian model can output one or more of the first rendering image of the global semantic feature associated with the target acquisition perspective parameter, the second rendering image of the spatial feature, and the third rendering image of the instance feature. Therefore, these rendering images can be used as the data basis for the robot to perform the object grasping task, so that the robot can accurately locate the object to be grasped and complete the object grasping task.

[0044] Figure 1A flow chart of a data processing method provided in an embodiment of the present disclosure is applicable to any scenario requiring three-dimensional scene understanding of a target area, such as a virtual reality scenario, an autonomous driving scenario, and a robot navigation scenario. The method can be executed by a data processing device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC, or a server, etc.

[0045] like Figure 1 As shown, the method of this embodiment may specifically include:

[0046] S110, obtaining target acquisition viewing angle parameters.

[0047] Among them, the target acquisition perspective parameter refers to the acquisition perspective parameter to be processed. The acquisition perspective parameter represents the shooting perspective information and position information of the perspective perception lens in three-dimensional space. It should also be noted that the acquisition perspective parameter can be represented by parameters of at least three dimensions. Preferably, the acquisition perspective parameter can be represented by parameters of six dimensions, wherein three parameters are used to represent the position information of the perspective perception lens, and three parameters are used to represent the rotation angle information of the perspective perception lens.

[0048] The data processing method provided in this embodiment can be integrated on any device with a perspective perception function. When the three-dimensional scene understanding task needs to be performed, the acquisition perspective parameter of the device is determined, and this acquisition perspective parameter is the target acquisition perspective parameter.

[0049] Exemplarily, the target acquisition viewing angle parameters can be expressed as (X, Y, Z, a, b, c), where the three parameters "X, Y, Z" represent the position coordinates of the viewing angle perception lens in three-dimensional space, and the three parameters "a, b, c" represent the rotation angle information of the viewing angle perception lens, that is, the rotation angle of the viewing angle perception lens relative to the spatial coordinate axis.

[0050] S120, inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter.

[0051] Among them, the target Gaussian model can be a pre-trained model, which is used to reconstruct the rendered image corresponding to the target area under a certain acquisition perspective parameter based on the acquisition perspective parameter. Since the target Gaussian model is a model obtained by training based on global semantic constraint information, spatial constraint information and instantiation constraint information, the rendered image output by the target Gaussian model includes not only a rendered image for representing the global semantic features associated with the target acquisition perspective parameter, but also a rendered image for representing the normal space information associated with the target acquisition perspective parameter. At the same time, in order to improve the execution efficiency of downstream tasks, the rendered image output by the target Gaussian model also includes a rendered image for representing the instance features associated with the target acquisition perspective parameter. The target area can be any spatial area, for example, the target area can be an office, a floor, etc. It can be understood that since the imaging range of the perspective perception lens under the target acquisition perspective parameter is certain, the imaging area associated with the target area under the target acquisition perspective parameter is a part of the target area. The rendered image refers to the feature map corresponding to the imaging area. In other words, the rendered image is a feature map obtained by the target Gaussian model by extracting features from the target acquisition perspective parameter. The number of rendered images includes multiple ones, and different rendered images represent feature information of different dimensions.

[0052] Specifically, the multiple rendered images include at least one or more of a first rendered image for representing global semantic features associated with the target acquisition viewing angle parameter, a second rendered image for spatial features, and a third rendered image for instance features.

[0053] The first rendered image refers to a feature map obtained by extracting global semantic features from the target acquisition viewing angle parameters. Among them, the global semantic features refer to the recognition and classification of objects in the scene, and the understanding of the relationship between objects and the semantic information of the scene. For example, different semantic information can be represented by different semantic coding information, and the first rendered image can be obtained by visually representing these semantic coding information. The global semantic features make the three-dimensional scene not just a stacking of geometric shapes, but contain rich semantic information, which can better understand and explain the objects in the three-dimensional scene and the relationship between them. Global semantic features can be applied in the case of object recognition and classification. For example, for different objects in the scene, such as tables, cups, books, etc., they are classified by different semantic coding information, which helps to accurately distinguish different objects in complex scenes. Global semantic features can also be applied in the case of understanding the relationship between objects. Global semantic features not only recognize individual objects, but also understand the relationship between objects, such as the spatial relationship and functional relationship between objects. For example, the relationship between vehicles and parking spaces in a parking lot can be identified through global semantic features, and the interaction between them can be understood. Global semantic features can also be used in scene understanding and interpretation. Global semantic features can be used to better understand the semantic content of the entire scene and explain events and activities in the scene.

[0054] The second rendered image refers to a feature map obtained by extracting spatial information features from the target acquisition viewing angle parameters. Among them, spatial information refers to information describing spatial geometry. For example, spatial information can be normal direction information of objects in the scene. Different normal direction information can be represented by different normal encoding information, and the second rendered image can be obtained by visually representing these normal encoding information. Normal direction information can be applied to the illumination model, which refers to simulating the shadow and reflection effects of the object by calculating the relationship between the normal direction of the object surface and the direction of light. According to the different normal directions, the brightness of the object surface can be calculated to achieve a realistic rendering effect. Normal direction information can also be applied to the scene of culling faces. Culling faces refers to determining whether to render the face according to the normal direction of the object surface during the rendering process. For invisible faces, rendering efficiency can be improved by culling. Normal direction information can help determine which faces are visible, so as to cull them.

[0055] The third rendered image refers to a feature map obtained by extracting instance features from the target acquisition viewing angle parameters. Among them, instance features refer to information that not only focuses on the category information of the object, but also further distinguishes different individual objects under the same category. For example, different individual objects can be represented by different unique instance identification codes, and the third rendered image can be obtained by visually representing these unique identification codes. Instance features can be applied to scenarios where different individuals under the same category need to be distinguished, such as identifying the number of people in a photo or multiple independent targets in a video. For example, if there are three apples in an image, all the pixels of each apple can be marked as "apple" through instance features, and each apple will be assigned a different unique instance identification code or color, so that you can know that there are three different apple instances in the image.

[0056] Optionally, the multiple rendered images may also include a fourth rendered image for characterizing color features associated with the target acquisition viewing angle parameters. The fourth rendered image refers to a feature map obtained by extracting color features of the target acquisition viewing angle parameters. Among them, the color features include information of the three color channels of red, green, and blue, and can provide richer visual details and color information. This information is very important for training the model because the model can improve its recognition and classification accuracy by learning these color and texture features. It should be noted that whether the fourth rendered image is included in the multiple rendered images is related to the specific actual business and is not specifically limited in this embodiment.

[0057] In this embodiment, environmental images under various different acquisition viewing angle parameters can be collected in the target area in advance. The initial Gaussian model is iteratively trained through multiple environmental images, and the parameters in the initial Gaussian model are continuously optimized to finally obtain a trained target Gaussian model. The target acquisition viewing angle parameters can be processed based on the target Gaussian model in such a way that the target acquisition viewing angle parameters are used as the input of the target Gaussian model, and the target Gaussian model performs multi-dimensional feature extraction on the target acquisition viewing angle parameters, and outputs multiple rendered images of different dimensions corresponding to the imaging of the target area under the target acquisition viewing angle parameters.

[0058] Based on the above exemplary embodiment, a schematic diagram of multiple rendered images output by the target Gaussian model is shown in Figure 2 .like Figure 2 As shown, the target acquisition viewing angle parameters (X, Y, Z, a, b, c) can be input into the target Gaussian model, and the target Gaussian model can output at least one of a first rendered image corresponding to a global semantic feature dimension, a second rendered image corresponding to a spatial feature dimension, and a third rendered image corresponding to an instance feature dimension by processing the target acquisition viewing angle parameters. Optionally, the target Gaussian model can also output a fourth rendered image corresponding to a color feature dimension.

[0059] The technical solution of the disclosed embodiment is applied to the execution of a three-dimensional scene understanding task, by obtaining a target acquisition viewing angle parameter, inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model for multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter, wherein the plurality of rendered images at least include one or more of a first rendered image for representing a global semantic feature associated with the target acquisition viewing angle parameter, a second rendered image for a spatial feature, and a third rendered image for an instance feature. The technical solution of the disclosed embodiment processes the target acquisition viewing angle parameter through a target Gaussian model, and can obtain a rendered image including a global semantic feature, a rendered image including a spatial feature, and a rendered image including an instance feature. These feature information provide three-dimensional spatial constraint information, improve the integrity of the object feature expression, and thereby execute the three-dimensional scene understanding task based on these rendered images, thereby ensuring execution accuracy and reliability.

[0060] Figure 3 A flowchart of another data processing method provided by an embodiment of the present disclosure. The technical solution of this embodiment is based on the above embodiment, and S120 is further described. The specific implementation method can refer to the detailed description of the embodiment of the present disclosure. Among them, the technical features that are the same or similar to the above embodiment are not repeated here.

[0061] like Figure 3 As shown, the method of this embodiment may specifically include:

[0062] S210: Obtain target acquisition viewing angle parameters.

[0063] S220, inputting the target acquisition perspective parameters into the target Gaussian model to reconstruct a first rendered image including global semantic features and a second rendered image including spatial features under the target acquisition perspective based on the first module in the target Gaussian model; and reconstructing a third rendered image based on the first module and the second module in the target Gaussian model.

[0064] In this embodiment, the target Gaussian model includes a first module and a second module, the first module includes Gaussian geometry parameters and semantic feature extraction parameters, and the second module includes instance feature extraction parameters. Gaussian geometry parameters refer to Gaussian model parameters used to determine spatial feature information and color feature information. Semantic feature extraction parameters refer to Gaussian model parameters used to determine global semantic features. Instance feature extraction parameters refer to Gaussian model parameters used to determine instance features. In particular, Gaussian geometry parameters, semantic feature extraction parameters, and instance feature extraction parameters are all parameters determined by training the initial Gaussian model based on a large number of training samples.

[0065] Specifically, the schematic diagram of the process of the target Gaussian model processing the target acquisition viewing angle parameters is shown in Figure 4 .like Figure 4 As shown, the target acquisition viewing angle parameters can be input into the target Gaussian model, so that the imaging area image under the target acquisition viewing angle parameters can be determined. Furthermore, by processing the imaging area image through the semantic feature extraction parameters in the first module, a first rendering image including global semantic features corresponding to the imaging area image can be reconstructed, and by processing the imaging area image through the Gaussian geometry parameters in the first module, a second rendering image including spatial features corresponding to the imaging area image can be reconstructed. At the same time, the imaging area image can be processed through the Gaussian geometry parameters and semantic feature extraction parameters in the first module and the instance feature extraction parameters in the second module, and a third rendering image including instance features corresponding to the imaging area image can be reconstructed. In this embodiment, the target acquisition viewing angle parameters are processed and analyzed respectively through the pre-trained multi-level parameter information in the target Gaussian model, so that rendering images of multiple different dimensions can be obtained quickly and accurately.

[0066] In this embodiment, in order to adapt to the situation of extracting features from unscaled objects, multiple target Gaussian models can be pre-trained. Different Gaussian models are determined based on global sample semantic feature images with different precision parameters.

[0067] The global sample semantic feature images are sample images used to train the initial Gaussian model, and these sample images contain global semantic feature information. When the global sample semantic feature images are predetermined, different precision parameters can be used to mask the objects presented in the same original sample image to obtain object mask images of each object under different object extraction precision parameters, thereby constructing global sample semantic feature images of different object extraction scales according to different object mask images.

[0068] Optionally, the number of target Gaussian models may include one or more. In the case where the number of target Gaussian models is multiple, there are certain differences in the accuracy of the rendered images output by the target Gaussian models. The main reason for the difference in accuracy is that the accuracy of the sample images used when training different target Gaussian models is different.

[0069] In the case of including multiple target Gaussian models, a specific implementation method of determining multiple rendered images associated with the target area under the target acquisition viewing angle parameters may include: inputting the target acquisition viewing angle parameters into at least two Gaussian models respectively, so that each Gaussian model outputs multiple rendered images associated with the target area under the target acquisition viewing angle parameters.

[0070] Among them, the accuracy parameters of the multiple rendered images are related to the accuracy parameters of the global sample semantic feature image used in model training.

[0071] Specifically, global sample semantic feature images corresponding to different precision parameters can be pre-constructed, and initial Gaussian models can be trained based on various types of global sample semantic feature images to obtain multiple target Gaussian models. Therefore, on the basis of obtaining the target acquisition viewing angle parameters, the target acquisition viewing angle parameters can be input into each target Gaussian model respectively, and each Gaussian model can output multiple rendered images associated with the target area under the target acquisition viewing angle parameters. The advantage of this setting is that by determining the target Gaussian models corresponding to different precision parameters, features of objects of different scales can be extracted, and both global information corresponding to large-scale objects and layout detail information corresponding to small-scale objects can be extracted, thereby improving the accuracy and comprehensiveness of the object information extracted by the target Gaussian model.

[0072] It is understandable that target Gaussian models of different precisions can output multiple rendered images. When executing a specific target task, the rendered image output by each target Gaussian model can be analyzed to determine the rendering image with which precision to execute the target task, so as to improve the execution accuracy of the target task.

[0073] According to the technical solution of the embodiment of the present disclosure, after obtaining the target acquisition perspective parameter, the target acquisition perspective parameter can be input into the target Gaussian model, so as to reconstruct the first rendering image including the global semantic features and the second rendering image including the spatial features under the target acquisition perspective based on the first module in the target Gaussian model; and the third rendering image can be reconstructed based on the first module and the second module in the target Gaussian model, and the target acquisition perspective parameter can be processed and analyzed respectively by the pre-trained multi-level parameter information in the target Gaussian model, so that multiple rendering images of different dimensions can be obtained quickly and accurately. It is also possible to pre-train target Gaussian models corresponding to multiple different precision parameters, and when determining multiple rendering images associated with the target area under the target acquisition perspective parameter, the target acquisition perspective parameter can be input into at least two Gaussian models respectively, so that each Gaussian model outputs multiple rendering images associated with the target area under the target acquisition perspective parameter. By determining the target Gaussian models corresponding to different precision parameters, features of objects of different scales can be extracted, and both global information corresponding to large-scale objects and layout detail information corresponding to small-scale objects can be extracted, thereby improving the accuracy and comprehensiveness of object information extracted by the target Gaussian model. Furthermore, when a target task is specifically executed, the rendered image output by each target Gaussian model may be analyzed to determine the accuracy of the rendered image to be used to execute the target task, so as to improve the execution accuracy of the target task.

[0074] Figure 5 A flowchart of another data processing method provided by an embodiment of the present disclosure. The technical solution of this embodiment is based on the above embodiment, and S110 is further described. The specific implementation method can refer to the detailed description of the embodiment of the present disclosure. Among them, the technical features that are the same or similar to the above embodiment are not repeated here.

[0075] like Figure 5 As shown, the method of this embodiment may specifically include:

[0076] S310: When it is detected that the task execution conditions are met, a target acquisition viewing angle parameter is obtained based on a device corresponding to the application scenario.

[0077] The task execution condition is a preset condition used to trigger the start of execution of a certain task.

[0078] In this embodiment, the application scenarios to which the data processing method is applicable include: at least one of a virtual reality scenario, an autonomous driving scenario, and a robot navigation scenario.

[0079] In this embodiment, the virtual reality scene refers to a virtual environment that is similar to or completely new to the real environment. When the data processing method provided in this embodiment is applied to a virtual reality scene, the device corresponding to the application scene may be a target server with a scene simulation function. When instruction information for executing a virtual simulation task of a specific scene is detected, the task execution condition is met, and any viewing angle parameter corresponding to the virtual camera device may be determined as a target acquisition viewing angle parameter, and the target server may obtain the target acquisition viewing angle parameter.

[0080] An autonomous driving scenario refers to a process in which an autonomous driving vehicle interacts with other vehicles, infrastructure, weather, lighting, obstacles and other elements in the driving environment within a certain time and space range. In the case where the data processing method provided in this embodiment is applied to an autonomous driving scenario, the device corresponding to the application scenario may be an autonomous driving vehicle. When instruction information is detected that requires the autonomous driving vehicle to start executing an autonomous driving task, the task execution conditions are met at this time, and the current viewing angle parameters corresponding to the viewing angle perception device (e.g., a monocular camera, a binocular stereo camera, a panoramic camera, and an infrared camera, etc.) configured on the vehicle can be determined as the target acquisition viewing angle parameters, so that the autonomous driving vehicle can obtain the target acquisition viewing angle parameters.

[0081] Robot navigation scenarios refer to the processes and scenarios in which robots move autonomously and complete tasks in a specific environment. Robot navigation involves multiple key technologies and components, including positioning, path planning, obstacle avoidance, motion control, etc. These technologies and components work together to enable robots to effectively complete tasks in complex environments. In the case where the data processing method provided in this embodiment is applied to the target robot navigation scenario, the device corresponding to the application scenario may be a robot. When it is detected that the target robot needs to start executing task execution instructions such as object grasping tasks, item delivery tasks, target inspection tasks, etc., the task execution conditions are met at this time, and the current viewing angle parameters corresponding to the viewing angle perception device configured on the robot can be determined as the target acquisition viewing angle parameters, so that the robot can obtain the target acquisition viewing angle parameters.

[0082] S320, inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter.

[0083] The multiple rendered images include at least one or more of a first rendered image for representing global semantic features associated with the target acquisition viewing angle parameter, a second rendered image for spatial features, and a third rendered image for instance features. Optionally, the multiple rendered images may also include a fourth rendered image for representing color features associated with the target acquisition viewing angle parameter.

[0084] S330, calling at least one target image from the plurality of rendered images based on the target task, and determining a target result corresponding to the target task based on the at least one target image, so as to control the device to execute the target task corresponding to the target result.

[0085] The target task is a task to be executed corresponding to the task execution condition. The target image is an image among the multiple rendered images. The target result is the data content based on which the device executes the target task.

[0086] Specifically, the data processing method provided in this embodiment can be integrated on devices corresponding to various application scenarios. The dimensional information corresponding to the image required for the device to perform the target task is determined. When multiple rendered images associated with the target area under the target acquisition perspective parameters are obtained according to the target acquisition perspective parameters and the target Gaussian model, the rendered image under the corresponding dimension can be retrieved from the multiple rendered images as the target image according to the dimensional information corresponding to the image required for the target task. Thus, the device can process and analyze at least one target image according to the pre-configured downstream task execution method to obtain a target result corresponding to the target task. Thus, the device can execute the corresponding target task based on the target result.

[0087] For example, in the case of a virtual reality scene, the target task may be a virtual simulation task to reconstruct a specific object in a scene. Such tasks usually require a rendered image in the color feature dimension. In addition, such virtual simulation tasks also need to determine the spatial result and relative position information of a specific object. Such tasks also require a rendered image in the spatial feature dimension. Therefore, the second rendered image representing the spatial feature and the fourth rendered image representing the color feature in multiple rendered images can be determined as the target image. Then, the virtual simulation screen of the specific physical object is reconstructed according to the target image. These virtual simulation screens are the target results, and then, these virtual simulation screens can be displayed in the target server.

[0088] In the case of an autonomous driving scenario, the target task may be a path planning task in autonomous driving. Such path planning tasks usually require real-time path planning, so a rendered image of the spatial feature dimension is required; it is also necessary to identify the obstacle type, obstacle location, and individual obstacle information on the road in real time, so a rendered image of the global semantic feature dimension and a rendered image of the instance feature dimension are required. At this time, the target image is a first rendered image representing the global semantic feature, a second rendered image representing the spatial feature, and a third rendered image representing the instance feature in multiple rendered images. In this scenario, when multiple rendered images are obtained and the target image is determined, the obstacle information can be determined based on the target image, so that the obstacle information and the current position information of the autonomous driving vehicle can be fed back to the path planning module of the autonomous driving vehicle, so that the path planning module can plan the target path corresponding to the path planning task (the target result corresponding to the path planning task). Furthermore, the autonomous driving vehicle can realize the function of autonomous driving according to the target path generated in real time.

[0089] In the case of robot navigation scenes, the target task may be to perform an object grasping task. Such tasks usually require real-time path planning, so a rendered image of the spatial feature dimension is required; it is also necessary to identify information such as object type, object position, and individual object, so a rendered image of the global semantic feature dimension and a rendered image of the instance feature dimension are required. Therefore, the target image may include a first rendered image representing global semantic features, a second rendered image representing spatial features, and a third rendered image representing instance features in multiple rendered images. Furthermore, these target images can be fed back to the robot path planning module, and the robot path planning module can determine the target planning path corresponding to the robot (the target result corresponding to the object grasping task) according to the target image. Furthermore, the robot can perform the object grasping task according to the target planning path.

[0090] The technical solution of the disclosed embodiment has a wide range of application scenarios and can be applied in virtual reality scenarios, autonomous driving scenarios or robot navigation scenarios. In the application of virtual reality scenarios, the target image can help the target server identify and understand objects and scenes in the real world, thereby achieving more accurate superposition information and providing a more immersive experience. In the application of autonomous driving scenarios, the target image can provide rich geometry, shape and scale information to help autonomous driving vehicles more accurately perceive the surrounding environment, identify pedestrians, vehicles and other obstacles, thereby improving driving safety; it can enhance the accuracy of positioning and navigation, especially in complex and dynamic environments, and can provide more reliable environmental information to help vehicles make correct decisions; it can also optimize path planning and obstacle detection. Autonomous driving vehicles can reconstruct a three-dimensional model of the surrounding environment in real time for accurate path planning and obstacle detection, thereby improving driving safety and robustness. In the application of robot navigation scenarios, robots need to understand three-dimensional scenes to perform various tasks. Through multi-dimensional target images, robots can accurately implement operations such as obstacle avoidance and path planning. Robots can better understand the surrounding environment and make smarter decisions.

[0091] Figure 6 A flowchart of another data processing method provided by an embodiment of the present disclosure. The technical solution of this embodiment is based on the above embodiment, and the training process of the target Gaussian model is further described. The specific implementation method can refer to the detailed description of the embodiment of the present disclosure. Among them, the technical features that are the same or similar to the above embodiment are not repeated here.

[0092] like Figure 6 As shown, the method of this embodiment may specifically include:

[0093] S410: Acquire images to be processed under multiple camera perspectives associated with the target area.

[0094] The image to be processed is an original image used to represent the object contained in the target area under a certain camera viewing angle. The camera viewing angle refers to the image range or angle that can be captured by the camera lens.

[0095] In this embodiment, a camera device may be used to collect images to be processed in the target area in advance under the same camera perspective. For example, in order to achieve the convenience of shooting images, the camera device may be fixedly mounted on a movable and rotatable sphere, and by rotating the sphere, images to be processed under multiple camera perspectives may be obtained.

[0096] In this embodiment, there are multiple ways to obtain the images to be processed under multiple camera perspectives associated with the target area. The first way can be: after the camera device captures each image to be processed under a camera perspective in the target area, it stores it in the cache space, and after obtaining the images to be processed under multiple camera perspectives, these images to be processed are acquired at one time. Another processing method can be that the camera device communicates with the corresponding electronic device, and the electronic device can execute the solution provided in the embodiment of the present disclosure, send the image to be processed captured by the camera device to the electronic device, and then the image to be processed can be acquired based on the electronic device.

[0097] S420: For the image to be processed, based on the image to be processed and the multi-level context perception model, determine a global sample semantic feature map corresponding to the image to be processed.

[0098] Among them, the multi-level context-aware model includes an image feature extraction submodel, an object segmentation submodel and a pooling submodel. The image feature extraction submodel is used to extract the first semantic feature map of the image to be processed, the object segmentation submodel is used to process the object mask presented in the image to be processed to obtain the object mask image, and the pooling submodel is used to fuse the first semantic feature map and the object mask image to obtain a global sample semantic feature map. Among them, the first semantic feature map is the semantic feature map corresponding to the image to be processed. The semantic feature map refers to a graphical representation of the semantic features of the vocabulary or syntactic structure in the language, which displays the semantic features of the vocabulary or syntactic structure in a graphical way. The object mask image is used to specify the operation of a specific area in the image without affecting other parts. The mask is usually a binary or Boolean image of the same size as the original image, in which the selected area is marked as 1 (or True) and the remaining areas are marked as 0 (or False)1.

[0099] In this embodiment, the schematic diagram of the implementation process of determining the global sample semantic feature map corresponding to the image to be processed based on the multi-level context perception model is shown in Figure 7 .like Figure 7As shown, on the basis of obtaining the images to be processed corresponding to multiple camera perspectives, the processing process for each image to be processed is consistent, and any one of the images to be processed is used as an example for explanation. In the specific implementation process, the image to be processed can be input into the object segmentation sub-model, and the object segmentation sub-model can mask the object presented in the image to be processed and output the object mask image; at the same time, the image to be processed can be input into the image feature extraction sub-model, and the image feature extraction sub-model extracts the semantic feature information in the processed image and constructs a first semantic feature map based on these semantic feature information. Furthermore, the object mask image and the first semantic feature map can be input into the pooling sub-model, and the pooling sub-model fuses the object mask image and the first semantic feature map to obtain a global sample semantic feature map corresponding to the image to be processed.

[0100] Specifically, the specific implementation process of the pooling sub-model performing pooling processing on the object mask image and the first semantic feature map can be: the first semantic feature map can be expressed as Where D represents the number of channels, H represents the image height, and W represents the image width. Each object mask image under any object extraction accuracy parameter contains multiple masks The pooling sub-model can perform pooling processing on the object mask image and the first semantic feature map through the pooling processing formula, and the pooling processing formula can be expressed as:

[0101]

[0102] Where, L lang,h (v) represents the final generated global sample semantic feature map, where v represents different pixels.

[0103] S430, based on the image to be processed, the camera perspective of the image to be processed and the global sample semantic feature map of the image to be processed, determine the training samples that participate in the training to obtain the target Gaussian model.

[0104] In this embodiment, for the image to be processed under each camera perspective, the image to be processed, the camera perspective corresponding to the image to be processed, and the global sample semantic feature map of the image to be processed can be determined as a training sample, so that multiple training samples can be obtained. On this basis, these training samples can be used to iteratively train the initial Gaussian model to obtain the target Gaussian model.

[0105] In this embodiment, the image to be processed is processed by a multi-level context-aware model to extract image-level features, and is combined with a multi-level mask for semantic enhancement. The global sample semantic feature map obtained by the multi-level context-aware model is used to train the target Gaussian model. This enables the target Gaussian model to enhance the semantic expression capability of low-texture areas and complex structure objects, and improves the recognition capability and semantic expression accuracy of the target Gaussian model for different texture areas in complex scenes.

[0106] On the basis of the above embodiments, optionally, a specific implementation method of determining a global sample semantic feature map corresponding to the image to be processed based on the image to be processed and the multi-level context-aware model may include: when inputting the image to be processed into the multi-level context-aware model, masking the objects presented in the image to be processed based on at least two object extraction accuracy parameters set in the object segmentation sub-model to obtain an object mask image of each object under at least two object extraction accuracy parameters; inputting the object mask image and the first semantic feature map into a pooling sub-model to pool at least one object mask image and the first semantic feature map under different object extraction accuracy parameters based on the pooling sub-model to obtain a global sample semantic feature map corresponding to different object extraction accuracy parameters.

[0107] Among them, the object extraction accuracy parameter refers to the scale information based on which the object information is extracted from the object to be processed when processing the image to be processed. For example, the higher the object extraction accuracy parameter, the more the obtained object mask focuses on local information; the lower the object extraction accuracy parameter, the more the obtained object mask focuses on global information. Exemplarily, the object in the image to be processed includes a table. When the object extraction accuracy parameter is relatively low, the object mask image corresponding to the table is the entire table; when the object extraction accuracy parameter is relatively high, the object mask image corresponding to the table includes a desktop mask image, a first table leg mask image, a second table leg mask image, and the like.

[0108] In this embodiment, a plurality of different object extraction precision parameters can be pre-set in the object segmentation sub-model in the multi-level context-aware model. After the image to be processed is input into the object segmentation sub-model, the objects presented in the image to be processed can be masked according to the various object extraction precision parameters in the object segmentation sub-model, so that the object mask image corresponding to each object under various object extraction precision parameters can be obtained. For example, the object mask image can include an extremely fine precision mask map, a precision mask map, and a coarse precision mask map, which are expressed as

[0109] Based on this, the training samples under different precision parameters can be obtained by classifying the images to be processed, the camera perspective of the images to be processed and the global sample semantic feature maps of the images to be processed corresponding to the same object extraction precision parameters, so as to obtain the target Gaussian model based on the training samples under the same precision parameters.

[0110] Exemplarily, for the images to be processed corresponding to N different camera perspectives, N first global sample semantic feature images corresponding to the coarse-grained object extraction accuracy parameter; N second global sample semantic feature images corresponding to the fine-grained object extraction accuracy parameter; and N third global sample semantic feature images corresponding to the extremely fine-grained object extraction accuracy parameter can be determined. Thus, the N camera perspectives, the N images to be processed, and the N first global sample semantic feature images can be divided into one category to obtain the first training sample corresponding to the coarse-grained object extraction accuracy parameter; the N camera perspectives, the N images to be processed, and the N second global sample semantic feature images can be divided into one category to obtain the second training sample corresponding to the fine-grained object extraction accuracy parameter; the N camera perspectives, the N images to be processed, and the N third global sample semantic feature images can be divided into one category to obtain the third training sample corresponding to the extremely fine-grained object extraction accuracy parameter. Furthermore, the first training sample can be used to train the initial Gaussian model to obtain a first target Gaussian model corresponding to the coarse-grained object extraction precision parameter; the second training sample can be used to train the initial Gaussian model to obtain a second target Gaussian model corresponding to the fine-grained object extraction precision parameter; the third training sample can be used to train the initial Gaussian model to obtain a third target Gaussian model corresponding to the ultra-fine-grained object extraction precision parameter. In this embodiment, by determining the global sample semantic feature map corresponding to different object extraction precision parameters, constructing training samples with different precision parameters, and then using the training samples corresponding to different precision parameters to train target Gaussian models with multiple different feature extraction scales, the accuracy and comprehensiveness of the target Gaussian model in extracting object information is improved.

[0111] S440: After preliminarily adjusting the Gaussian geometric parameters in the Gaussian model to be trained based on the training samples, adjust the Gaussian geometric parameters and semantic feature extraction parameters based on the training samples.

[0112] The Gaussian model to be trained refers to an initial Gaussian model whose model parameters are default parameters. The Gaussian model to be trained includes Gaussian geometric parameters and semantic feature extraction parameters.

[0113] In this embodiment, the Gaussian model to be trained can be initially adjusted and trained by the training samples to perform preliminary adjustment on the Gaussian geometric parameters in the Gaussian model to be trained. Subsequently, the Gaussian model to be trained can be trained for a second round by the training samples to synchronously adjust the Gaussian geometric parameters and semantic feature extraction parameters in the Gaussian model to be trained.

[0114] Optionally, a specific implementation method of performing preliminary adjustment on the Gaussian geometric parameters in the Gaussian model to be trained based on the samples to be trained may include: inputting the camera perspective in the training samples into the Gaussian model to be trained, outputting multiple predicted images, and determining the two-dimensional image loss based on the two-dimensional images in the multiple predicted images and the images to be processed in the training samples, as well as the scaling coefficient loss of the Gaussian points in the Gaussian model to be trained; performing preliminary adjustment on the Gaussian geometric parameters in the Gaussian model to be trained based on the two-dimensional image loss and the scaling coefficient loss; and obtaining the Gaussian geometric parameters after preliminary adjustment when it is detected that the convergence conditions are met.

[0115] The predicted image is a two-dimensional color image obtained by imaging prediction based on the camera perspective by the Gaussian model to be trained. The scaling factor of the Gaussian point refers to a coefficient used to adjust the weight of the integral point in the Gaussian quadrature method. In the Gaussian quadrature method, the integral points are usually called Gaussian points. These points are used to approximate the integral value in the integral. The role of the scaling factor is to appropriately scale the weight of the Gaussian point to ensure the stability and accuracy of the calculation.

[0116] In this embodiment, the processing process for each training sample is consistent. In order to clearly introduce the present technical solution, one of the training samples may be used as an example. The camera perspective in the training sample may be input into the Gaussian model to be trained, and the Gaussian model to be trained may output multiple predicted images corresponding to the camera perspective. Furthermore, the two-dimensional image in the multiple predicted images and the image to be processed in the training sample are brought into the two-dimensional image loss function to obtain the two-dimensional image loss. The formula of the two-dimensional image loss function can be expressed as:

[0117]

[0118] In the formula, C i represents the predicted image, I i represents the image to be processed in the training sample, represents the 2D image loss;

[0119] At the same time, the scaling factor loss can be determined according to the scaling factor of the Gaussian point in the Gaussian model to be trained. The scaling factor loss can be expressed as:

[0120]

[0121] In the formula, {s 1 ,s 2 ,s 3}=diag(S i ), {s 1 ,s 2 ,s 3} refers to the scaling factor of the Gaussian points.

[0122] On this basis, the Gaussian geometric parameters in the Gaussian model to be trained are preliminarily adjusted through the two-dimensional image loss and the scaling factor loss. For example, the training error of the two-dimensional image loss and the scaling factor loss, that is, the loss parameter, can be used as a detection of whether the loss function has reached the convergence condition, such as whether the training error is less than the preset error or whether the error change trend is stable, or whether the current number of iterations is equal to the preset number. If the detection reaches the convergence condition, such as the training error of the loss function is less than the preset error or the error change tends to be stable, it indicates that the Gaussian geometric parameters have been adjusted, and the iterative training can be stopped at this time. If it is detected that the convergence condition is not reached at present, the Gaussian model to be trained can be further trained according to the training samples until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, it indicates that the Gaussian geometric parameters have been adjusted, and the iterative training can be stopped at this time.

[0123] In this embodiment, the Gaussian geometric parameters in the Gaussian model to be trained are preliminarily adjusted through two-dimensional image loss and scaling coefficient loss to determine the accuracy of the predicted image output by the Gaussian model, thereby providing accurate dependent data for subsequent training processes.

[0124] Optionally, a specific implementation method for adjusting Gaussian geometric parameters and semantic feature extraction parameters based on training samples may include: inputting the training samples into the Gaussian model to be trained after preliminary adjustment of the Gaussian geometric parameters, and outputting multiple predicted images; determining a normal loss value based on a predicted normal image in the predicted image and a theoretical normal image corresponding to the image to be processed in the training sample; and determining a semantic loss value based on predicted semantic feature images of different objects in the predicted image and a global sample semantic feature map in the training sample; and adjusting the Gaussian geometric parameters and semantic feature extraction parameters that have been preliminary adjusted based on the normal loss value and the semantic loss value.

[0125] Among them, the predicted normal image refers to the graphical representation corresponding to the normal direction information in the predicted image. The theoretical normal image refers to the graphical representation corresponding to the normal direction information in the image to be processed. When the image to be processed is obtained, the corresponding theoretical normal image can be determined. The predicted semantic feature image refers to the semantic feature map predicted by the Gaussian model to be trained based on the predicted image.

[0126] In this embodiment, the Gaussian model to be trained after the preliminary adjustment of Gaussian geometric parameters can be called the first Gaussian model to be trained. The camera perspective in the training sample can be input into the first Gaussian model to be trained again, and the first Gaussian model to be trained can output multiple predicted images.

[0127] Furthermore, the predicted normal image corresponding to each predicted image can be determined; thus, the predicted normal image in the predicted image and the theoretical normal image corresponding to the image to be processed in the training sample are brought into the normal loss function to obtain the normal loss value. The formula of the normal loss function can be expressed as:

[0128]

[0129] Where n render Represents the predicted normal image in the predicted image, n gt represents the theoretical normal image corresponding to the image to be processed in the training sample, Represents the normal loss value.

[0130] The purpose of this setting is to constrain the geometric distribution of Gaussian points in the optimized Gaussian model by determining the normal loss value, and to ensure that the language feature information can be accurately aligned with the object surface through multi-view normal constraints and spatial consistency optimization, so that the semantic features have higher geometric consistency in three-dimensional space, so as to improve the spatial prediction ability of the target Gaussian model, thereby improving the accuracy of the segmentation results and query results when performing three-dimensional scene understanding tasks based on the target Gaussian model.

[0131] In this embodiment, the first semantic loss value can be determined by using L2 semantic supervision to determine the predicted semantic feature image based on different objects in the predicted image and the global sample semantic feature map in the training sample. At the same time, in terms of semantic supervision, on the basis of conventional L2 semantic supervision, a semantic grouping strategy is also adopted to minimize the semantic distance of rendered semantic features of different pixels in the same mask, so that the semantics are consistent within the same mask and the features within the semantic grouping are similar. Figure 8 , the semantic grouping strategy can be characterized by the following semantic feature close loss function, and the formula of the semantic feature close loss function can be expressed as:

[0132]

[0133] Where M is the mask of the image to be processed under a certain object extraction accuracy parameter The total number of and They respectively represent the semantic feature values ​​corresponding to different pixel points in the same object mask of the predicted semantic feature image. Indicates the semantic feature pull-in loss value.

[0134] On this basis, the semantic loss value Can be the first semantic loss value Close the loss value with semantic features The superposition result.

[0135] Based on the above embodiments, optionally, the method also includes: determining the predicted Gaussian points corresponding to the predicted semantic feature image, and determining the associated Gaussian points associated with the predicted Gaussian points; for at least one predicted Gaussian point, determining the spatial Gaussian point loss based on the feature information of the predicted Gaussian point and the associated Gaussian point in the predicted semantic feature map; and updating the semantic loss value based on the spatial Gaussian point loss.

[0136] The predicted Gaussian point is the three-dimensional Gaussian point corresponding to the predicted semantic feature image in the Gaussian model to be trained. The associated Gaussian point is the Top-K nearest three-dimensional Gaussian point corresponding to the predicted Gaussian point. For each predicted Gaussian point, it is necessary to determine multiple associated Gaussian points that are adapted to it. The spatial Gaussian point loss refers to the loss value of the semantic feature value of the three-dimensional predicted Gaussian point corresponding to the predicted semantic feature image.

[0137] In this embodiment, the schematic diagram of spatial Gaussian point loss is shown in Fig. 9 The purpose of setting the spatial Gaussian point loss is to add spatial constraints to the semantic feature information, align the semantic features with the top k nearest Gaussian distributions by utilizing KL divergence supervision to reduce the spatial outliers in the semantic three-dimensional Gaussian field. This strategy enables the semantic embedding surface field to be accurately integrated into the scene surface and achieve comprehensive scene understanding.

[0138] In this embodiment, the spatial Gaussian point loss can be determined by the following spatial Gaussian point loss function, which can be expressed as:

[0139]

[0140] Among them, N represents the number of predicted Gaussian points, N k is the number of associated Gaussian points of the jth predicted Gaussian point. Refers to the semantic feature value corresponding to the j-th predicted Gaussian point, Refers to the semantic feature value of the associated Gaussian point of the j-th predicted Gaussian point, represents the spatial Gaussian point loss.

[0141] On this basis, the semantic loss value Can be the first semantic loss value Semantic features close the loss value And spatial Gaussian point loss The superposition result.

[0142] In this embodiment, the Gaussian geometry parameters and semantic feature extraction parameters in the first Gaussian model to be trained are adjusted by the normal loss value and the semantic loss value. Similarly, the training errors of the normal loss value and the semantic loss value, that is, the loss parameters, can be used to detect whether the loss function currently reaches the convergence condition. If the convergence condition is reached, for example, the training error of the loss function reaches less than the preset error or the error change tends to be stable, it indicates that the Gaussian geometry parameters and the semantic feature extraction parameters have been adjusted.

[0143] S450: After the Gaussian geometric parameters and the semantic feature extraction parameters are adjusted, the instance feature extraction parameters in the Gaussian model to be trained are initialized based on the semantic feature extraction parameters, so as to adjust the instance feature extraction parameters based on the training samples.

[0144] In this embodiment, the Gaussian model to be trained also includes instance feature extraction parameters. After the Gaussian geometric parameters and semantic feature extraction parameters are adjusted, the instance feature extraction parameters in the Gaussian model to be trained can be initialized by the semantic feature extraction parameters. Further, the instance feature extraction parameters in the Gaussian model to be trained can be adjusted by the training samples.

[0145] Optionally, a specific implementation method for adjusting instance feature extraction parameters based on training samples may include: inputting the camera perspective in the training samples into the Gaussian noise model to be trained after instantiating the instance feature extraction parameters, and outputting multiple predicted images; determining the average instance feature corresponding to each object mask image based on the predicted instantiated image in the predicted image, the theoretical instantiated image corresponding to the image to be processed in the training samples, and each object mask image in the image to be processed; determining the instantiation loss based on multiple average instance features of the image to be processed, so as to adjust the instance feature extraction parameters based on the instantiation loss.

[0146] Among them, the predicted instantiation image is the image representation corresponding to the instance label information in the predicted image. The theoretical instantiation image refers to the image representation corresponding to the instance label information in the image to be processed. When the image to be processed is obtained, the corresponding theoretical instantiation image can be determined. The average instance feature refers to the average semantic feature value corresponding to a certain instance mask area. The instantiation loss represents the difference between the instantiation semantic feature values ​​of the predicted instantiation image and the theoretical instantiation image.

[0147] In this embodiment, after initializing the instance feature extraction parameters in the Gaussian model to be trained based on the semantic feature extraction parameters, the Gaussian noise model to be trained is obtained. Then, the camera perspective in the training sample can be input into the Gaussian noise model to be trained, and the Gaussian noise model to be trained can output a predicted image; a predicted instantiation image corresponding to each predicted image can be determined; for each object mask image in the image to be processed, the average instance feature corresponding to each object mask image is determined by the average instance feature function, and the average instance feature function can be expressed as:

[0148]

[0149] In the formula, Represents the semantic feature value corresponding to each pixel in the object mask image, M i represents the i-th object mask image, represents the average instance feature corresponding to the i-th object mask image.

[0150] Furthermore, a first instantiation loss may be determined based on a predicted instantiation image in the predicted image and a theoretical instantiation image corresponding to the image to be processed in the training sample; thus, the average instance feature may be updated based on the first instantiation loss.

[0151] Optionally, the instantiation loss can be determined based on multiple average instance features of the image to be processed and an instantiation loss function. For a schematic diagram of the instantiation loss, see Fig.10 , the instantiated loss function can be expressed as:

[0152]

[0153] In the formula, represents the average instance feature corresponding to the j-th object mask image, represents the average instance feature corresponding to the kth object mask image, D min Represents the minimum value of the average instance feature difference between every two object mask images. represents the instantiation loss.

[0154] In this embodiment, the instance feature extraction parameters can be adjusted according to the instantiation loss, and the instance feature extraction parameters in the Gaussian noise model to be trained can be adjusted by the instantiation loss. Similarly, the training error of the instantiation loss, that is, the loss parameter, can be used as a detection of whether the loss function currently reaches the convergence condition. If the detection reaches the convergence condition, such as the training error of the loss function reaches less than the preset error or the error change tends to be stable, it indicates that the instance feature extraction parameters have been adjusted. In this embodiment, the average instance features of different mask images in the instantiated image are calculated and predicted, and the instance feature information of different objects is maximized through the instantiation loss, thereby improving the instance partitioning capability of the target Gaussian model.

[0155] S460: The Gaussian model to be trained obtained when the iteration condition is met is used as the target Gaussian model.

[0156] In this embodiment, the iteration condition is a pre-set trigger condition for the end of training of the Gaussian model to be trained. For example, the iteration condition may be that all loss functions converge, or that the current iteration number reaches a preset iteration number threshold. When the iteration condition is detected to be met, the Gaussian model to be trained at this time may be used as the target Gaussian model.

[0157] S470: Obtain target acquisition viewing angle parameters.

[0158] S480. Input the target acquisition viewing angle parameters into a pre-trained target Gaussian model for multi-dimensional feature extraction, and output a plurality of rendered images associated with the target area under the target acquisition viewing angle parameters, wherein the plurality of rendered images include at least one or more of a first rendered image for representing global semantic features associated with the target acquisition viewing angle parameters, a second rendered image for spatial features, and a third rendered image for instance features.

[0159] The technical solution of the embodiment of the present disclosure is that when constructing training samples, the image to be processed is processed by a multi-level context-aware model, image-level features can be extracted, and semantic enhancement is performed by combining a multi-level mask. The global sample semantic feature map obtained by the multi-level context-aware model is used to train the target Gaussian model through the global sample semantic feature map, so that the target Gaussian model has the ability to enhance the semantic expression of low-texture areas and complex structure objects, and improves the recognition ability and semantic expression accuracy of the target Gaussian model for different texture areas in complex scenes. Therefore, the target Gaussian model is pre-trained according to the training samples, and then the target acquisition perspective parameter is obtained, and the target acquisition perspective parameter is input into the pre-trained target Gaussian model for multi-dimensional feature extraction, and multiple rendered images associated with the target area under the target acquisition perspective parameter are output, wherein the multiple rendered images at least include one or more of the first rendered image for representing the global semantic feature associated with the target acquisition perspective parameter, the second rendered image of the spatial feature, and the third rendered image of the instance feature, thereby improving the accuracy and convenience of determining the multi-dimensional rendered image based on the target Gaussian model.

[0160] Fig.11 A structural diagram of a data processing device provided by an embodiment of the present disclosure is shown in FIG. Fig.11 As shown, the device includes: a viewing angle parameter acquisition module 510 and a rendering image determination module 520.

[0161] The viewing angle parameter acquisition module 510 is used to acquire the viewing angle parameters of the target acquisition;

[0162] A rendering image determination module 520 is used to input the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and output a plurality of rendering images associated with the target area under the target acquisition viewing angle parameter;

[0163] The multiple rendered images include at least one or more of a first rendered image for representing global semantic features associated with target acquisition viewing angle parameters, a second rendered image for spatial features, and a third rendered image for instance features.

[0164] On the basis of the above technical solution, optionally, the rendering image determination module 520 is specifically used to input the target acquisition perspective parameters into the target Gaussian model, so as to reconstruct a first rendered image including global semantic features and a second rendered image including spatial features under the target acquisition perspective based on the first module in the target Gaussian model; and to reconstruct the third rendered image based on the first module and the second module in the target Gaussian model; wherein the first module includes Gaussian geometry parameters and semantic feature extraction parameters, and the second module includes instance feature extraction parameters.

[0165] On the basis of the above-mentioned optional technical solutions, in the case of including at least two Gaussian models, optionally, the rendering image determination module 520 is further specifically used to input the target acquisition viewing angle parameters into the at least two Gaussian models respectively, so that each Gaussian model outputs a plurality of rendering images associated with the target area under the target acquisition viewing angle parameters; wherein the at least two Gaussian models are determined based on global sample semantic feature images with different precision parameters, and the precision parameters of the plurality of rendering images are related to the precision parameters of the global sample semantic feature images used in model training.

[0166] Based on the above-mentioned optional technical solutions, optionally, the application scenarios of the method include: at least one of a virtual reality scenario, an autonomous driving scenario, and a robot navigation scenario.

[0167] On the basis of the above technical solution, optionally, the viewing angle parameter acquisition module 510 is specifically used to acquire the target acquisition viewing angle parameter based on the device corresponding to the application scenario when it is detected that the task execution condition is met;

[0168] Correspondingly, the data processing device also includes: a device control module, which is used to call at least one target image among the multiple rendering images based on the target task, and determine the target result corresponding to the target task based on the at least one target image, so as to control the device to perform the target task corresponding to the target result.

[0169] Based on the above optional technical solutions, optionally, the data processing device further includes: a training sample determination module; the training sample determination module includes:

[0170] A to-be-processed image acquisition unit, used to acquire to-be-processed images under multiple camera viewing angles associated with the target area;

[0171] A semantic feature map determining unit, configured to determine, for the image to be processed, a global sample semantic feature map corresponding to the image to be processed based on the image to be processed and a multi-level context perception model;

[0172] A training sample determination unit, used to determine the training samples involved in the training to obtain the target Gaussian model based on the image to be processed, the camera angle of view of the image to be processed and the global sample semantic feature map of the image to be processed;

[0173] Among them, the multi-level context-aware model includes an image feature extraction submodel, an object segmentation submodel and a pooling submodel. The image feature extraction submodel is used to extract the first semantic feature map of the image to be processed, the object segmentation submodel is used to process the object mask presented in the image to be processed to obtain the object mask image, and the pooling submodel is used to fuse the first semantic feature map and the object mask image to obtain the global sample semantic feature map.

[0174] Based on the above optional technical solutions, optionally, the semantic feature map determining unit includes:

[0175] an object mask image determination subunit, configured to, when inputting the image to be processed into the multi-level context-aware model, perform mask processing on the objects presented in the image to be processed based on at least two object extraction accuracy parameters set in the object segmentation submodel, so as to obtain an object mask image of each object under the at least two object extraction accuracy parameters;

[0176] The semantic feature map determination subunit is used to input the object mask image and the first semantic feature map into the pooling sub-model, so as to pool at least one object mask image and the first semantic feature map under different object extraction accuracy parameters based on the pooling sub-model to obtain a global sample semantic feature map corresponding to different object extraction accuracy parameters.

[0177] On the basis of the above-mentioned optional technical solutions, optionally, the training sample determination module is also used to obtain training samples under different precision parameters by classifying the images to be processed corresponding to the same object extraction precision parameters, the camera perspective of the images to be processed and the global sample semantic feature map of the images to be processed, so as to obtain the target Gaussian model based on the training samples under the same precision parameters.

[0178] Based on the above optional technical solutions, optionally, the data processing device further includes: a target model training module; the target model training module includes:

[0179] A first parameter adjustment unit, configured to adjust the Gaussian geometric parameters and the semantic feature extraction parameters based on the training samples after preliminarily adjusting the Gaussian geometric parameters in the Gaussian model to be trained based on the training samples;

[0180] A second parameter adjustment unit is used to initialize the instance feature extraction parameters in the Gaussian model to be trained based on the semantic feature extraction parameters after the Gaussian geometric parameters and the semantic feature extraction parameters are adjusted, so as to adjust the instance feature extraction parameters based on the training sample;

[0181] The target model determination unit is used to use the Gaussian model to be trained obtained when the iteration condition is met as the target Gaussian model.

[0182] Based on the above optional technical solutions, optionally, the first parameter adjustment unit includes:

[0183] A geometric parameter adjustment subunit is used to input the camera perspective in the training sample into the Gaussian model to be trained, output a plurality of predicted images, and determine the two-dimensional image loss based on the two-dimensional image in the plurality of predicted images and the image to be processed in the training sample, as well as the scaling coefficient loss of the Gaussian point in the Gaussian model to be trained; perform preliminary adjustment on the Gaussian geometric parameters in the Gaussian model to be trained based on the two-dimensional image loss and the scaling coefficient loss; and obtain the Gaussian geometric parameters after preliminary adjustment when it is detected that the convergence condition is met.

[0184] The semantic feature extraction parameter subunit is used to input the training sample into the Gaussian model to be trained after the Gaussian geometric parameters are preliminarily adjusted, and output multiple predicted images; determine the normal loss value based on the predicted normal image in the predicted image and the theoretical normal image corresponding to the image to be processed in the training sample; and determine the semantic loss value based on the predicted semantic feature images of different objects in the predicted image and the global sample semantic feature map in the training sample; adjust the preliminarily adjusted Gaussian geometric parameters and the semantic feature extraction parameters based on the normal loss value and the semantic loss value.

[0185] The semantic feature extraction parameter subunit is also used to determine the predicted Gaussian point corresponding to the predicted semantic feature image, and determine the associated Gaussian point associated with the predicted Gaussian point; for the at least one predicted Gaussian point, based on the feature information of the predicted Gaussian point and the associated Gaussian point in the predicted semantic feature image, determine the spatial Gaussian point loss; and update the semantic loss value based on the spatial Gaussian point loss.

[0186] Based on the above optional technical solutions, optionally, the second parameter adjustment unit includes:

[0187] A predicted image determination subunit, used for inputting the camera view angle in the training sample into the Gaussian noise model to be trained after instantiating the instance feature extraction parameters, and outputting a plurality of predicted images;

[0188] an instance feature determination subunit, configured to determine an average instance feature corresponding to each object mask image according to the predicted instantiation image in the predicted image, the theoretical instantiation image corresponding to the image to be processed in the training sample, and each object mask image in the image to be processed;

[0189] The instantiation parameter adjustment subunit is used to determine the instantiation loss based on multiple average instance features of the image to be processed, so as to adjust the instance feature extraction parameter based on the instantiation loss.

[0190] The technical solution of the disclosed embodiment is applied to the execution of a three-dimensional scene understanding task, by obtaining a target acquisition viewing angle parameter, inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model for multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter, wherein the plurality of rendered images at least include one or more of a first rendered image for representing a global semantic feature associated with the target acquisition viewing angle parameter, a second rendered image for a spatial feature, and a third rendered image for an instance feature. The technical solution of the disclosed embodiment processes the target acquisition viewing angle parameter through a target Gaussian model, and can obtain a first rendered image including a global semantic feature, a second rendered image including a spatial feature, and a third rendered image including an instance feature, which information provides three-dimensional spatial constraint information and improves the integrity of the object feature expression, thereby executing a three-dimensional scene understanding task based on these rendered images, and ensuring execution accuracy and reliability.

[0191] The data processing device provided in the embodiments of the present disclosure can execute the data processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0192] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0193] Fig.12 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.12 , which shows an electronic device (eg, Fig.12 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0194] like Fig.12As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An edit / output (I / O) interface 605 is also connected to the bus 604.

[0195] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0196] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0197] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0198] The electronic device provided in the embodiment of the present disclosure and the data processing method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the present embodiment has the same beneficial effects as the above embodiment.

[0199] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the data processing method provided by the above embodiments is implemented.

[0200] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0201] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0202] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0203] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0204] Get the target acquisition viewing angle parameters;

[0205] Inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter;

[0206] The multiple rendered images include at least one or more of a first rendered image for representing global semantic features associated with target acquisition viewing angle parameters, a second rendered image for spatial features, and a third rendered image for instance features.

[0207] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0208] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0209] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0210] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0211] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0212] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0213] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0214] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data processing method, characterized in that: include: Get the target acquisition viewing angle parameters; Inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter; The multiple rendered images include at least one or more of a first rendered image for representing global semantic features associated with target acquisition viewing angle parameters, a second rendered image for spatial features, and a third rendered image for instance features.

2. The method according to claim 1, characterized in that The step of inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter, comprises: Inputting the target acquisition viewing angle parameter into the target Gaussian model, so as to reconstruct a first rendered image including global semantic features and a second rendered image including spatial features under the target acquisition viewing angle based on the first module in the target Gaussian model; and reconstructing the third rendered image based on the first module and the second module in the target Gaussian model; The first module includes Gaussian geometry parameters and semantic feature extraction parameters, and the second module includes instance feature extraction parameters.

3. The method according to claim 1, characterized in that In the case of including at least two target Gaussian models, inputting the target acquisition viewing angle parameter into the pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendered images associated with the target area under the target acquisition viewing angle parameter, comprises: Inputting the target acquisition viewing angle parameters into the at least two target Gaussian models respectively, so that each target Gaussian model outputs a plurality of rendered images associated with the target area under the target acquisition viewing angle parameters; Among them, the at least two Gaussian models are determined based on global sample semantic feature images with different precision parameters, and the precision parameters of the multiple rendered images are related to the precision parameters of the global sample semantic feature images used in model training.

4. The method according to any one of claims 1 to 3, characterized in that: The application scenarios of the method include: at least one of a virtual reality scenario, an autonomous driving scenario, and a robot navigation scenario.

5. The method according to claim 4, characterized in that The step of obtaining target acquisition viewing angle parameters includes: When it is detected that the task execution condition is met, acquiring the target acquisition viewing angle parameter based on a device corresponding to the application scenario; Accordingly, after obtaining the plurality of rendered images, the method further includes: At least one target image among the plurality of rendered images is retrieved based on a target task, and a target result corresponding to the target task is determined based on the at least one target image, so as to control the device to execute the target task corresponding to the target result.

6. The method according to claim 1, characterized in that The method further comprises: Acquire images to be processed under multiple camera perspectives associated with the target area; For the image to be processed, based on the image to be processed and a multi-level context perception model, determining a global sample semantic feature map corresponding to the image to be processed; Determining training samples for training to obtain the target Gaussian model based on the image to be processed, the camera perspective of the image to be processed, and the global sample semantic feature map of the image to be processed; Among them, the multi-level context-aware model includes an image feature extraction submodel, an object segmentation submodel and a pooling submodel. The image feature extraction submodel is used to extract the first semantic feature map of the image to be processed, the object segmentation submodel is used to process the object mask presented in the image to be processed to obtain the object mask image, and the pooling submodel is used to fuse the first semantic feature map and the object mask image to obtain the global sample semantic feature map.

7. The method according to claim 6, characterized in that The step of determining a global sample semantic feature map corresponding to the image to be processed based on the image to be processed and the multi-level context perception model includes: When the image to be processed is input into the multi-level context-aware model, mask processing is performed on the objects presented in the image to be processed based on at least two object extraction accuracy parameters set in the object segmentation sub-model to obtain an object mask image of each object under the at least two object extraction accuracy parameters; The object mask image and the first semantic feature map are input into the pooling sub-model, so as to pool at least one object mask image and the first semantic feature map under different object extraction accuracy parameters based on the pooling sub-model to obtain a global sample semantic feature map corresponding to different object extraction accuracy parameters.

8. The method according to claim 7, characterized in that The method further comprises: By classifying and processing the image to be processed corresponding to the same object extraction precision parameter, the camera perspective of the image to be processed and the global sample semantic feature map of the image to be processed, training samples under different precision parameters are obtained, and the target Gaussian model is obtained by training based on the training samples under the same precision parameter.

9. The method according to claim 6, characterized in that After obtaining the training sample, the method further includes: After preliminarily adjusting the Gaussian geometric parameters in the Gaussian model to be trained based on the training samples, adjusting the Gaussian geometric parameters and the semantic feature extraction parameters based on the training samples; After the Gaussian geometric parameters and the semantic feature extraction parameters are adjusted, initializing instance feature extraction parameters in the Gaussian model to be trained based on the semantic feature extraction parameters, so as to adjust the instance feature extraction parameters based on the training samples; The Gaussian model to be trained obtained when the iteration condition is met is used as the target Gaussian model.

10. The method according to claim 9, characterized in that Based on the samples to be trained, the Gaussian geometric parameters in the Gaussian model to be trained are initially adjusted, including: Inputting the camera view in the training sample into the Gaussian model to be trained, outputting a plurality of predicted images, and determining the two-dimensional image loss based on the two-dimensional image in the plurality of predicted images and the image to be processed in the training sample, and determining the scaling coefficient loss of the Gaussian point in the Gaussian model to be trained; Preliminarily adjusting the Gaussian geometry parameters in the Gaussian model to be trained based on the two-dimensional image loss and the scaling factor loss; When it is detected that the convergence condition is met, the preliminarily adjusted Gaussian geometric parameters are obtained.

11. The method according to claim 9, characterized in that The adjusting the Gaussian geometry parameters and the semantic feature extraction parameters based on the training sample includes: Inputting the training samples into the Gaussian model to be trained after preliminary adjustment of Gaussian geometric parameters, and outputting a plurality of predicted images; Determining a normal loss value based on a predicted normal image in the predicted image and a theoretical normal image in the training sample corresponding to the image to be processed; and Determining a semantic loss value based on the predicted semantic feature images of different objects in the predicted image and the global sample semantic feature map in the training sample; Based on the normal loss value and the semantic loss value, the initially adjusted Gaussian geometry parameters and the semantic feature extraction parameters are adjusted.

12. The method according to claim 11, characterized in that The method further comprises: Determining a predicted Gaussian point corresponding to the predicted semantic feature image, and determining an associated Gaussian point associated with the predicted Gaussian point; For the at least one predicted Gaussian point, determining a spatial Gaussian point loss based on feature information of the predicted Gaussian point and the associated Gaussian point in the predicted semantic feature map; The semantic loss value is updated based on the spatial Gaussian point loss.

13. The method according to claim 9, characterized in that: The adjusting the instance feature extraction parameters based on the training sample includes: Inputting the camera view angle in the training sample into the Gaussian noise model to be trained after instantiating the instance feature extraction parameters, and outputting a plurality of predicted images; Determining an average instance feature corresponding to each object mask image according to the predicted instantiation image in the predicted image, the theoretical instantiation image corresponding to the image to be processed in the training sample, and each object mask image in the image to be processed; Based on a plurality of average instance features of the image to be processed, an instantiation loss is determined to adjust the instance feature extraction parameters based on the instantiation loss.

14. A data processing device, characterized in that: include: A viewing angle parameter acquisition module is used to obtain the viewing angle parameters of the target acquisition; A rendering image determination module, used for inputting the target acquisition viewing angle parameter into a pre-trained target Gaussian model to perform multi-dimensional feature extraction, and outputting a plurality of rendering images associated with the target area under the target acquisition viewing angle parameter; The multiple rendered images include at least one or more of a first rendered image for representing global semantic features associated with target acquisition viewing angle parameters, a second rendered image for spatial features, and a third rendered image for instance features.

15. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 13.

16. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the data processing method according to any one of claims 1 to 13.

Images