Virtual scene processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a virtual scene processing method and device, electronic equipment, a computer readable storage medium and a computer program product, and the method comprises the steps: carrying out the data preprocessing of multi-modal image data and illumination attribute data of a to-be-processed virtual scene, and obtaining a multi-modal illumination feature and an illumination attribute feature; determining a reconstruction error of the to-be-processed virtual scene based on the multi-modal illumination features; performing error analysis on the reconstruction error by adopting a detection threshold value corresponding to the to-be-processed virtual scene to obtain an illumination detection result of the to-be-processed virtual scene; determining an illumination prediction result of the to-be-processed virtual scene based on the illumination attribute features; and determining an illumination processing strategy of the to-be-processed virtual scene based on the illumination detection result and the illumination prediction result. According to the method and the device, the accurate illumination detection result and the illumination prediction result can be obtained based on the reconstruction error and the illumination attribute characteristics, so that the illumination processing strategy for effectively solving the illumination anomaly problem of the to-be-processed virtual scene is obtained.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a virtual scene processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Lighting in virtual scenes, a crucial rendering technique, greatly enhances the realism of rendered scenes and is widely used in various games. Furthermore, appropriately adding lighting effects to game scenes can effectively enhance the atmosphere and provide a better player experience. However, during game play, various factors inevitably lead to areas where the lighting distribution within the game scene does not match expectations. This is known as lighting anomalies, for example, due to improper light source position, intensity, or direction.

[0003] In related technologies, the accuracy and real-time detection performance of lighting anomaly detection in virtual scenes are often limited, resulting in the inability to efficiently, accurately and in real time detect lighting anomalies and provide developers with targeted lighting optimization suggestions, which seriously affects the visual quality and realism of virtual scenes. Summary of the Invention

[0004] The embodiments of the present application provide a virtual scene processing method, device, electronic device, computer-readable storage medium and computer program product, which can obtain accurate lighting detection results and lighting prediction results based on reconstruction errors and lighting attribute characteristics, and thus obtain a lighting processing strategy that effectively solves the lighting anomaly problem of the virtual scene to be processed.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a virtual scene processing method, the method comprising: performing data preprocessing on multimodal image data and illumination attribute data of a virtual scene to be processed, respectively, to obtain multimodal illumination features and illumination attribute features correspondingly; determining a reconstruction error of the virtual scene to be processed based on the multimodal illumination features; performing error analysis on the reconstruction error using a detection threshold corresponding to the virtual scene to be processed, to obtain an illumination detection result of the virtual scene to be processed; determining an illumination prediction result of the virtual scene to be processed based on the illumination attribute features; and determining an illumination processing strategy for the virtual scene to be processed based on the illumination detection result and the illumination prediction result.

[0007] An embodiment of the present application provides a virtual scene processing device, comprising: a data preprocessing module, configured to perform data preprocessing on multimodal image data and illumination attribute data of a virtual scene to be processed, respectively, to obtain multimodal illumination features and illumination attribute features correspondingly; a reconstruction error determination module, configured to determine a reconstruction error of the virtual scene to be processed based on the multimodal illumination features; an error analysis module, configured to perform error analysis on the reconstruction error using a detection threshold corresponding to the virtual scene to be processed, to obtain an illumination detection result of the virtual scene to be processed; a lighting prediction result determination module, configured to determine an illumination prediction result of the virtual scene to be processed based on the illumination attribute features; and a lighting processing strategy determination module, configured to determine an illumination processing strategy for the virtual scene to be processed based on the illumination detection result and the illumination prediction result.

[0008] In some embodiments, the reconstruction error determination module is further used to: perform feature reconstruction on the multimodal illumination feature to obtain a multimodal reconstruction feature; and determine the reconstruction error of the virtual scene to be processed based on the multimodal illumination feature and the multimodal reconstruction feature.

[0009] In some embodiments, the reconstruction error determination module is also used to: encode the multimodal illumination features through the encoding module of the pre-trained virtual scene processing model to obtain the encoded multimodal illumination features; and decode the encoded multimodal illumination features through the decoding module of the pre-trained virtual scene processing model to obtain the multimodal reconstruction features.

[0010] In some embodiments, the reconstruction error determination module is further used to: obtain the eigenvalues of each dimension in the multimodal illumination vector corresponding to the multimodal illumination feature and the eigenvalues of each dimension in the multimodal reconstruction vector corresponding to the multimodal reconstruction feature; the multimodal illumination vector has the same dimension as the multimodal reconstruction vector; determine the difference between the eigenvalues of each dimension in the multimodal illumination vector and the eigenvalues of the corresponding dimension in the multimodal reconstruction vector, and determine the sum of the squares of the differences; and determine the reconstruction error of the virtual scene to be processed based on the sum of the squares.

[0011] In some embodiments, the device also includes a detection threshold determination module, and the detection threshold determination is used to: obtain multiple scene object data of the virtual scene to be processed and the scene weight corresponding to each of the scene object data; integrate the products of each of the scene object data and the scene weight respectively to obtain multiple integration results; obtain the integration weight corresponding to each of the integration results; add the products of the multiple integration results and the integration weight to obtain the detection threshold corresponding to the virtual scene to be processed.

[0012] In some embodiments, the error analysis module is also used to: when the reconstruction error is less than or equal to the detection threshold, determine that the lighting detection result of the virtual scene to be processed is normal lighting; when the reconstruction error is greater than the detection threshold, determine that the lighting detection result of the virtual scene to be processed is abnormal lighting.

[0013] In some embodiments, the lighting prediction result determination module is also used to: encode the lighting attribute features through the encoding module of the pre-trained virtual scene processing model to obtain the encoded lighting attribute features; and perform feature mapping on the encoded lighting attribute features through the feature mapping module of the pre-trained virtual scene processing model to obtain the lighting prediction result of the virtual scene to be processed.

[0014] In some embodiments, the data preprocessing module is further used to: perform normalization processing on the multimodal image data and the illumination attribute data respectively, and obtain normalized multimodal image data and normalized illumination attribute data accordingly; perform feature extraction on the normalized multimodal image data and the normalized illumination attribute data respectively, and obtain multimodal feature vectors and the illumination attribute features accordingly; perform feature fusion on the multimodal feature vectors to obtain the multimodal illumination features.

[0015] In some embodiments, the lighting processing strategy determination module is also used to: when the lighting detection result is a lighting abnormality, perform parameter abnormality analysis on the lighting prediction result to obtain the lighting parameters to be adjusted under the virtual scene to be processed; based on the lighting parameters to be adjusted, determine the lighting processing strategy for the virtual scene to be processed.

[0016] In some embodiments, the illumination detection result and the illumination prediction result are realized by a pre-trained virtual scene processing model; the device also includes a model training module, and the model training module is used to: obtain sample data, the sample data including multimodal image sample data and illumination attribute sample data; the illumination attribute sample data is the illumination attribute information of at least one branch task associated with the illumination detection task; the multimodal image sample data and the illumination attribute sample data are respectively preprocessed to obtain multimodal illumination sample features and illumination attribute sample features, and the multimodal illumination sample features and the illumination attribute sample features are input into the virtual scene processing model; the encoding module of the virtual scene processing model is used to encode the multimodal illumination sample features and the illumination attribute sample features to obtain the encoded multimodal illumination sample features. sample features and encoded illumination attribute sample features; decoding the encoded multimodal illumination sample features through the decoding module of the virtual scene processing model to obtain multimodal reconstruction sample features, and feature mapping the encoded illumination attribute sample features through the feature mapping module of the virtual scene processing model to obtain illumination sample prediction results; constructing the sample reconstruction error of the illumination detection task based on the multimodal illumination sample features and the multimodal reconstruction sample features; determining the branch loss of each branch task based on the illumination sample prediction results and the labels of the illumination attribute sample data; determining the loss result of the virtual scene processing model based on the sample reconstruction error and the branch loss; updating the model parameters of the virtual scene processing model based on the loss result to obtain the pre-trained virtual scene processing model.

[0017] In some embodiments, the model training module is also used to: obtain the weight of each branch loss in the at least one branch loss; based on the weight, perform weighted processing on each branch loss to obtain the weighted loss of each branch task; based on the reconstruction error and the weighted loss, construct the model loss of the virtual scene processing model; perform loss calculation on the model loss to obtain the loss result of the virtual scene processing model.

[0018] In some embodiments, the model training module is also used to: determine the parameter gradient of the model parameters of the virtual scene processing model; determine the model deletion parameters of the virtual scene processing model based on the parameter gradient and a preset gradient threshold; perform parameter deletion on the model parameters based on the model deletion parameters to obtain the model update parameters of the virtual scene processing model; update the model update parameters of the virtual scene processing model based on the loss result to obtain the pre-trained virtual scene processing model.

[0019] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions; and a processor for implementing the virtual scene processing method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0020] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the virtual scene processing method provided in the embodiment of the present application when executed by a processor.

[0021] An embodiment of the present application provides a computer program product, which includes computer-executable instructions, and the computer-executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, the virtual scene processing method provided in the embodiment of the present application is implemented.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] First, the multimodal image data and lighting attribute data of the virtual scene to be processed are preprocessed separately. Based on the multimodal lighting characteristics obtained after data preprocessing, the reconstruction error of the virtual scene to be processed is determined. Then, using the detection threshold corresponding to the virtual scene to be processed, an error analysis is performed on the reconstruction error to obtain the lighting detection result for the virtual scene to be processed. In this way, the image data corresponding to multiple modalities in the multimodal image data is used as input information for the lighting detection process, fully utilizing the multimodal image information of the virtual scene to be processed. Based on the detection threshold corresponding to the virtual scene to be processed, a precise error analysis of the reconstruction error is performed, thereby obtaining accurate lighting detection results and improving the accuracy of lighting detection.

[0024] Secondly, based on the lighting attribute characteristics of the pre-processed data, the lighting prediction results for the virtual scene to be processed are determined. Furthermore, based on the lighting detection results and the lighting prediction results, a lighting treatment strategy for the virtual scene to be processed is determined. In this way, by combining accurate lighting detection and lighting prediction results, even if the lighting detection results indicate lighting anomalies, the lighting prediction results can be used to provide developers with targeted lighting treatment strategies, effectively resolving lighting anomalies in the virtual scene to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural diagram of the virtual scene processing system architecture provided by an embodiment of the present application;

[0026] Figure 2 is a structural diagram of a virtual scene processing device provided in an embodiment of the present application;

[0027] Figure 3 This is an optional flowchart of the virtual scene processing method provided in the embodiment of the present application;

[0028] Figure 4 This is another optional flowchart of the virtual scene processing method provided in the embodiment of the present application;

[0029] Figure 5 Schematic diagram of the process of pre-training virtual scene processing model training method provided in the embodiment of the present application;

[0030] Figure 6 This is a schematic diagram of the framework of multimodal intelligent lighting detection provided by an embodiment of the present application;

[0031] Figure 7 Schematic diagram of an RGB image of a real-time acquisition scene provided by an embodiment of the present application;

[0032] Figure 8 is a schematic diagram of a depth map of a real-time acquisition scene provided by an embodiment of the present application;

[0033] Figure 9 is a schematic diagram of a normal map of a real-time acquisition scene provided by an embodiment of the present application;

[0034] Figure 10 Schematic diagram of the network structure and network parameters of the pre-trained convolutional neural network provided in the embodiment of the present application;

[0035] Figure 11 is a scatter diagram of fusion features obtained through feature fusion operation provided in an embodiment of the present application;

[0036] Figure 12 This is a schematic diagram of the output of the fusion feature provided in the embodiment of the present application;

[0037] Figure 13 Schematic diagram of the structure of the encoding neural network provided in the embodiment of the present application;

[0038] Figure 14 Schematic diagram of the structure of the decoding neural network provided in the embodiment of the present application;

[0039] Figure 15 Schematic diagram of the unsupervised learning loss curve based on autoencoding provided in an embodiment of the present application;

[0040] Figure 16 Schematic diagram of the structure of the autoencoder multi-task network provided in an embodiment of the present application;

[0041] Figure 17 This is a schematic diagram of the specific process of implementing the model pruning technology provided in the embodiment of the present application;

[0042] Figure 18 This is a logical diagram of the dynamic adjustment of the reconstruction error threshold provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0044] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] If similar descriptions of "first / second" appear in the application documents, the following explanation is added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0046] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0047] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0048] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0049] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0050] 1) Lighting Anomaly Detection: Identify and detect areas in the game scene where the lighting distribution does not match the expected one. Lighting anomalies may include improper light source position, intensity, or direction, as well as missing occluded objects.

[0051] 2) Multimodal input: Using multiple types of data as input to the neural network. In the lighting anomaly detection task, in addition to using common RGB images, you can also consider using data from other modalities, such as depth maps and normal maps. This additional information can help improve detection accuracy.

[0052] 3) Depth map: A two-dimensional image that represents the distance of each pixel in the scene from the camera. The depth map provides geometric information about the scene and helps understand the relative positions of objects.

[0053] 4) Unsupervised learning: A method for training neural networks without labeled data. Unsupervised learning methods can automatically learn features and structures from input data, reducing the cost of labeling data.

[0054] 5) Convolutional Neural Networks: A special type of neural network used to process grid-like data such as images and videos. Convolutional Neural Networks (CNNs) have the characteristics of local receptive fields, weight sharing, and spatial invariance. They can effectively extract image features and perform tasks such as classification and detection.

[0055] 6) Game Graphics Rendering: The process of generating scene images within a game engine. Game graphics rendering involves calculations of scene geometry, lighting, materials, shadows, and other aspects, aiming to achieve high-quality, high-performance visual effects.

[0056] One related approach is to detect lighting anomalies based on traditional image processing. This approach uses traditional image processing methods, such as edge detection and histogram equalization, to analyze the lighting distribution in a scene. It determines whether lighting anomalies exist by calculating local or global statistical features of the image, such as brightness, contrast, and color distribution. However, its drawbacks include relying solely on RGB images and failing to utilize additional information such as depth maps and normal maps, which limits detection accuracy. Traditional image processing capabilities struggle to adapt to complex scenes and lighting variations. Manual adjustment of thresholds and parameters is required, resulting in poor generalization performance.

[0057] Another approach is a supervised learning-based CNN illumination anomaly detection solution. This solution uses a supervised learning CNN to detect illumination anomalies. The CNN model is trained with a large number of labeled images of scenes with normal and abnormal illumination to learn the characteristics of illumination anomalies. However, its disadvantages include the need for a large amount of labeled illumination anomaly data, which is costly; reliance on RGB images, without utilizing additional information such as depth maps and normal maps, results in poor detection accuracy; and high computational resource requirements, which limits real-time performance.

[0058] In summary, the related art has the following technical problems:

[0059] 1. It only relies on RGB images and does not use additional information such as depth maps and normal maps, resulting in poor detection accuracy;

[0060] 2. For complex scenes and lighting changes, thresholds and parameters need to be manually adjusted, resulting in poor generalization performance;

[0061] 3. The computing resource requirements are high and the real-time detection performance is poor.

[0062] Based on at least one of the above technical problems existing in the above-mentioned related technologies, the embodiment of the present application adopts multimodal input (RGB image, depth map and normal map), fully utilizes the color, geometry and lighting information of the scene, and can detect lighting anomalies more accurately than the solution that relies only on RGB images; an adaptive threshold and parameter adjustment method is designed to enable the model to automatically adjust the threshold and parameters of lighting anomaly detection according to the characteristics of the scene, thereby improving the generalization performance of the model in different scenarios; model optimization and acceleration technologies such as pruning, quantization, knowledge distillation, etc. are applied to improve the real-time performance of the model in the game engine and ensure smooth game operation; compared with the solution based on traditional image processing, the use of convolutional neural networks can better adapt to complex scenes and lighting changes, and improve the robustness of lighting anomaly detection. In summary, the embodiment of the present application has obvious advantages in detection accuracy, generalization performance, real-time performance and adaptability, and provides game developers with an efficient, accurate and real-time lighting anomaly detection solution.

[0063] The following describes an exemplary application of the virtual scene processing device (i.e., electronic device) provided by the embodiment of the present application. The device provided by the embodiment of the present application can be implemented as a laptop computer, tablet computer, desktop computer, set-top box, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), smart phone, smart speaker, smart watch, smart TV, vehicle-mounted terminal, and other types of user terminals capable of data processing or virtual scene processing, and can also be implemented as a server. Below, an exemplary application when the virtual scene processing device is implemented as a server will be described.

[0064] See also Figure 1 , Figure 1 It is a structural diagram of the virtual scene processing system 100 architecture provided in an embodiment of the present application. In order to support a virtual scene processing application, the virtual scene processing application is run on the terminal 400. The terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0065] The terminal 400 is used to send a virtual scene processing request to the server 200. The server 200 constitutes a virtual scene processing device of an embodiment of the present application. The server 200 is used to respond to the virtual scene processing request, perform data preprocessing on the multimodal image data and illumination attribute data of the virtual scene to be processed, and obtain corresponding multimodal illumination features and illumination attribute features; based on the multimodal illumination features, determine the reconstruction error of the virtual scene to be processed; use the detection threshold corresponding to the virtual scene to be processed to perform error analysis on the reconstruction error to obtain the illumination detection result of the virtual scene to be processed; based on the illumination attribute features, determine the illumination prediction result of the virtual scene to be processed; based on the illumination detection result and the illumination prediction result, determine the illumination processing strategy of the virtual scene to be processed. After obtaining the illumination processing strategy, the server 200 returns the illumination processing strategy to the terminal 400, so that the illumination processing strategy can be output at the terminal 400 or the terminal 400 can continue to perform the next business processing based on the illumination processing strategy.

[0066] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, car terminal, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0067] See also Figure 2 , Figure 2 is a schematic structural diagram of an electronic device 40 provided in an embodiment of the present application, Figure 2The electronic device 40 shown may be a virtual scene processing device, which includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the virtual scene processing device are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0068] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0069] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0070] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0071] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0072] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0073] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0074] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0075] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0076] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0077] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 A virtual scene processing device 455 stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a data preprocessing module 4551, a reconstruction error determination module 4552, an error analysis module 4553, an illumination prediction result determination module 4554, and an illumination processing strategy determination module 4555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0078] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the virtual scene processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0079] In some embodiments, the terminal or server can implement the virtual scene processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, or it can be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment and can be run. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0080] The virtual scene processing method provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the virtual scene processing method of each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.

[0081] See also Figure 3 , Figure 3 This is an optional flow chart of the virtual scene processing method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are described below, and the execution subject of the virtual scene processing method is a server as an example. The method includes the following steps S101 to S105:

[0082] In step S101 , data preprocessing is performed on the multimodal image data and the illumination attribute data of the virtual scene to be processed, respectively, to obtain multimodal illumination features and illumination attribute features accordingly.

[0083] In the embodiments of the present application, the virtual scene to be processed can be any scene constructed through three-dimensional modeling that requires lighting detection, for example, a game scene. The multimodal image data is image data of multiple modalities in the virtual scene to be processed, including image data of other modalities in addition to GRB image data, such as depth image data and normal image data.

[0084] RGB image data contains color information for the virtual scene being processed, depth image data contains distance relationship information between objects in the virtual scene, and normal image data contains surface normal vector information for objects in the virtual scene being processed. Lighting attribute data contains different types of information related to the lighting properties of the virtual scene being processed. Data preprocessing is used to normalize multimodal image data and lighting attribute data. For example, data preprocessing can be a standardization or normalization process, or it can be a feature extraction process for the data. Depending on the different virtual scene processing requirements, one or more data preprocessing methods can be selected to preprocess the input data.

[0085] Here, after the multimodal image data and lighting attribute data of the virtual scene to be processed undergo data preprocessing, multimodal lighting features and lighting attribute features are obtained. This data preprocessing improves the input data quality and reliability of the lighting detection and prediction processes, improves data distribution, and helps improve the efficiency and accuracy of the lighting detection and prediction processes.

[0086] Step S102: determining a reconstruction error of the virtual scene to be processed based on the multimodal illumination features.

[0087] In an embodiment of the present application, the reconstruction error refers to the difference or loss between the reconstructed output features and the original input features obtained by reconstructing (or restoring) the original input features through a pre-trained virtual scene processing model. The multimodal lighting features after data preprocessing here are the original input features, and the error between the output features of the pre-trained virtual scene processing model and the multimodal lighting features is the reconstruction error. The reconstruction error can be used to evaluate the ability of the pre-trained virtual scene processing model to reconstruct or restore the original input features. A smaller reconstruction error indicates that the model can accurately reconstruct the original input features, while a larger reconstruction error means that the model cannot completely restore the original input features. In the illumination detection process of an embodiment of the present application, the reconstruction error is used to determine whether the illumination in the virtual scene to be processed is abnormal by the size of the error between the multimodal illumination features and the output features of the pre-trained virtual scene processing model. The illumination detection process implemented using a pre-trained virtual scene processing model can be an unsupervised learning process. For example, the illumination detection process can be implemented using an autoencoder. The autoencoder first encodes the multimodal illumination features and then decodes them to obtain the decoded output features. Finally, the reconstruction error of the virtual scene to be processed is determined by combining the decoded output features with the multimodal illumination features. The autoencoder can be a convolutional autoencoder or a multi-layer autoencoder, and the type of autoencoder is not limited here. In other embodiments, the illumination detection process can also be implemented using a generative adversarial network.

[0088] Here, an unsupervised virtual scene processing model is used to construct the reconstruction error, reducing the dependence on a large amount of labeled illumination anomaly data, reducing the annotation cost, and facilitating the subsequent error analysis process of the reconstruction error to realize the automation of illumination detection.

[0089] Step S103 : Using a detection threshold corresponding to the virtual scene to be processed, an error analysis is performed on the reconstruction error to obtain a lighting detection result of the virtual scene to be processed.

[0090] In an embodiment of the present application, the detection threshold corresponding to the virtual scene to be processed is a dynamic threshold calculated based on the scene object data in the actual virtual scene to be processed, which is used to measure whether the lighting in the virtual scene to be processed is normal during the lighting detection process. The detection threshold is not a fixed threshold and can be dynamically adjusted according to changes in the virtual scene to be processed. The scene object data can be the number of scene objects in the virtual scene to be processed, for example, the number of scene objects, the number of triangles used to characterize whether the object rendering is complex, etc. If the virtual scene to be processed changes, the scene object data in the virtual scene to be processed will change accordingly, thereby changing the numerical value of the detection threshold.

[0091] Error analysis is the process of analyzing the reconstruction error using a detection threshold. The detection threshold is numerically compared with the reconstruction error, and the comparison result is used to determine the lighting detection result for the virtual scene being processed. The lighting detection result is the result obtained after performing a lighting test on the virtual scene being processed. The lighting detection results include normal lighting and abnormal lighting. When the reconstruction error is greater than the detection threshold, the lighting detection result is abnormal lighting; when the reconstruction error is less than or equal to the detection threshold, the lighting detection result is normal lighting.

[0092] In some embodiments, in order to determine the appropriate detection threshold, a validation dataset can also be used to evaluate the lighting detection performance (e.g., accuracy, recall rate, and F1 score, etc.) of the pre-trained virtual scene processing model under different detection thresholds, and select the detection threshold corresponding to the best lighting detection performance.

[0093] Here, according to the characteristics of the virtual scene to be processed, the detection threshold can be dynamically adjusted, that is, when the virtual scene to be processed becomes more complex or simpler, the detection threshold will also rise or fall accordingly, so that the embodiment of the present application can achieve efficient lighting detection under various different virtual scenes to be processed and lighting conditions, and improve the generalization performance of lighting detection in different virtual scenes to be processed.

[0094] Step S104: determining the illumination prediction result of the virtual scene to be processed based on the illumination attribute characteristics.

[0095] In the embodiments of the present application, the illumination prediction result refers to the illumination information of the virtual scene to be processed, which is predicted through the illumination prediction process of the pre-trained virtual scene processing model. For example, the light source position, illumination intensity, illumination direction, illumination distance, etc. can be predicted. It can be seen that the illumination prediction process can predict a variety of illumination information, and the type of illumination information predicted can be adjusted according to the different prediction requirements in the virtual scene to be processed. The illumination prediction process can be a multi-task learning process, and different prediction requirements can be regarded as different illumination-related branch tasks. The branch tasks can be used to construct a multi-task network, and the model parameters can be shared through the multi-task network, thereby improving the overall illumination prediction performance.

[0096] Step S105 : determining a lighting processing strategy for the virtual scene to be processed based on the lighting detection result and the lighting prediction result.

[0097] In the embodiment of the present application, the lighting processing strategy is used to achieve real-time optimization of lighting parameters in the virtual scene to be processed. When the lighting detection result of the virtual scene to be processed is normal, or when the lighting detection result of the virtual scene to be processed is abnormal, the lighting prediction results are analyzed to accurately locate the lighting parameters causing the abnormal lighting, that is, to obtain the abnormal lighting parameters. Then, a lighting processing strategy is formulated based on the abnormal lighting parameters to achieve adjustment of the abnormal lighting parameters. Furthermore, the virtual scene to be processed can be subjected to lighting detection again until the lighting detection result of the virtual scene to be processed is normal.

[0098] In some embodiments, the lighting prediction result can also be a probability value of whether the lighting parameters are abnormal, obtained through the lighting prediction process, and the probability value is between 0 and 1. That is, based on the lighting prediction result output by the pre-trained virtual scene processing model, the lighting abnormality parameters are directly determined, and then the lighting processing strategy for the virtual scene to be processed is determined based on the lighting abnormality parameters.

[0099] Here, accurate lighting detection results and lighting prediction results obtained according to the lighting detection process and the lighting prediction process can effectively achieve lighting optimization of the virtual scene to be processed, thereby improving the lighting effect and rendering quality of the virtual scene to be processed.

[0100] The virtual scene processing method provided in the embodiment of the present application, on the one hand, first performs data preprocessing on the multimodal image data and lighting attribute data of the virtual scene to be processed respectively, and determines the reconstruction error of the virtual scene to be processed based on the multimodal lighting characteristics after data preprocessing. Then, the reconstruction error is analyzed by the detection threshold corresponding to the virtual scene to be processed to obtain the lighting detection result of the virtual scene to be processed. In this way, the image data corresponding to multiple modes in the multimodal image data is used as the input information of the lighting detection process, and the image information of multiple modes in the virtual scene to be processed is fully utilized, and based on the detection threshold corresponding to the virtual scene to be processed, an accurate error analysis of the reconstruction error is achieved, thereby obtaining an accurate lighting detection result and improving the accuracy of lighting detection. On the other hand, based on the lighting attribute characteristics after data preprocessing, the lighting prediction result of the virtual scene to be processed is determined, and based on the lighting detection result and the lighting prediction result, the lighting processing strategy of the virtual scene to be processed is determined. In this way, by combining accurate lighting detection results and lighting prediction results, when the lighting detection results are lighting anomalies, targeted lighting processing strategies can be provided to developers based on the lighting prediction results, thereby effectively solving the lighting anomaly problem of the virtual scene to be processed.

[0101] The following will describe the virtual scene processing method in the embodiment of the present application in conjunction with the interaction between the terminal and the server in the virtual scene processing system. It should be noted that the virtual scene processing method here is a virtual scene processing method implemented by the interaction between the terminal and the server, which is essentially the same as the virtual scene processing method executed by the server in the above embodiment. The only difference is that the embodiment of the present application also describes the actions performed by the terminal during the execution of the virtual scene processing method, and some steps can be executed by both the terminal and the server. Therefore, for the steps in this embodiment that are the same as those in the above embodiment but have different execution entities, this embodiment is only an illustrative description. During the implementation process, they can be executed by any execution entity, and the embodiment of the present application does not limit this.

[0102] Figure 4 This is another optional flow chart of the virtual scene processing method provided in the embodiment of the present application, such as Figure 4 As shown, the method includes the following steps S201 to S215:

[0103] Step S201: The terminal receives a virtual scene processing operation input by a user.

[0104] In an embodiment of the present application, a user can input a virtual scene processing operation in the client of the virtual scene processing application. In the virtual scene processing application, a virtual scene processing function can be provided. The user (who can be a virtual scene processing designer and a virtual scene developer) can input a virtual scene processing operation in the virtual scene processing function page to trigger a virtual scene processing request.

[0105] In some embodiments, when a user inputs a virtual scene processing operation, they may also simultaneously input the multimodal image data and lighting attribute data of the virtual scene to be processed. When the terminal receives the multimodal image data and lighting attribute data, a confirmation virtual scene processing window will pop up on the virtual scene processing function page. After the terminal detects that the user has clicked the confirm virtual scene processing button, it will further perform lighting detection and lighting prediction on the multimodal image data and lighting attribute data to determine the lighting processing strategy for the virtual scene to be processed. Alternatively, in other embodiments, the user may directly input the multimodal image data and lighting attribute data on the virtual scene processing function page. When the terminal receives the multimodal image data and lighting attribute data, it may directly trigger the virtual scene processing function, perform further lighting detection and lighting prediction on the multimodal image data and lighting attribute data, and determine the lighting processing strategy for the virtual scene to be processed.

[0106] Step S202: The terminal generates a virtual scene processing request in response to the virtual scene processing operation.

[0107] In embodiments of the present application, user-entered data can be encapsulated into a virtual scene processing request. For example, on the display interface of a virtual scene processing application, multimodal image data and lighting attribute data of a virtual scene to be processed are displayed. The user can select or sample data based on actual needs, and then encapsulate the multimodal image data and lighting attribute data entered by the user into the virtual scene processing request. Alternatively, a detection threshold corresponding to the virtual scene to be processed entered by the user can be encapsulated into the virtual scene processing request.

[0108] Step S203: The terminal sends a virtual scene processing request to the server.

[0109] In step S204 , the server performs normalization processing on the multimodal image data and the illumination attribute data in response to the virtual scene processing request, and obtains normalized multimodal image data and normalized illumination attribute data respectively.

[0110] In an embodiment of the present application, in response to a virtual scene processing request, if the virtual scene processing request encapsulates multimodal image data and lighting attribute data of the virtual scene to be processed, the multimodal image data and lighting attribute data of the virtual scene to be processed can be directly parsed to obtain.

[0111] In an embodiment of the present application, normalization processing is used to limit the multimodal image data and illumination attribute data to a certain data range (for example, between 0 and 1 or between -1 and 1). The normalization processing method can be a normalization method based on maximum and minimum values, or a normalization method based on mean and standard deviation. For example, the normalization method based on mean and standard deviation can first calculate the mean of the multimodal image data and the illumination attribute data, as well as the standard deviation of the multimodal image data and the illumination attribute data, and then subtract the multimodal image data from the mean of the multimodal image data, divide the subtracted value by the standard deviation of the multimodal image data, and finally obtain the normalized multimodal image data. Similarly, subtract the illumination attribute data from the mean of the illumination attribute data, divide the subtracted value by the standard deviation of the illumination attribute data, and finally obtain the normalized illumination attribute data. There is no limitation on the normalization processing method here.

[0112] Here, by normalizing the multimodal image data and illumination attribute data separately, the normalized multimodal image data and illumination attribute data are scaled to a unified dimension, thereby improving the model accuracy and convergence speed in the subsequent illumination detection process and illumination prediction process.

[0113] In step S205 , the server performs feature extraction on the normalized multimodal image data and the normalized illumination attribute data, respectively, to obtain a multimodal feature vector and illumination attribute features.

[0114] In the embodiments of the present application, feature extraction refers to processing and analyzing the normalized multimodal image data and the normalized illumination property data using a feature extraction function, and extracting information that is less susceptible to random interference as features, that is, extracting the most effective features from the normalized multimodal image data and the normalized illumination property data. Feature extraction from the normalized multimodal image data and the normalized illumination property data can typically be performed using a neural network model, such as a VGG-16 network. The feature extraction method is not limited herein.

[0115] In step S206 , the server performs feature fusion on the multimodal feature vector to obtain a multimodal illumination feature.

[0116] In the embodiments of the present application, feature fusion refers to fusing feature vectors from different modalities in a multimodal feature vector. For example, feature fusion is performed on the feature vectors corresponding to the RGB image data, the feature vectors corresponding to the depth image data, and the feature vectors corresponding to the normal image data. Feature fusion can be feature addition or feature concatenation.

[0117] Here, through the feature fusion process, the different modal features in the multimodal feature vector are jointly modeled, so that the multimodal lighting features after feature fusion contain more lighting feature information of the virtual scene to be processed, and in the subsequent lighting detection process, they are used as input features of the pre-trained virtual scene processing model to improve the accuracy of lighting detection.

[0118] In step S207 , the server performs feature reconstruction on the multimodal illumination feature to obtain a multimodal reconstructed feature.

[0119] In an embodiment of the present application, the specific implementation process of feature reconstruction of multimodal illumination features is as follows: the multimodal illumination features are encoded by the encoding module of the pre-trained virtual scene processing model to obtain the encoded multimodal illumination features; and the encoded multimodal illumination features are decoded by the decoding module of the pre-trained virtual scene processing model to obtain the multimodal reconstruction features.

[0120] That is, the illumination detection process is implemented using a pre-trained virtual scene processing model, which includes an encoding module and a decoding module. The feature reconstruction process is implemented using the encoding module and decoding module of the pre-trained virtual scene processing model. The encoding module is used to learn the implicit features of the multimodal illumination features through the encoding process, using the multimodal illumination features as the original input features. This can be represented by an encoding function h = f(x), where x represents the original input features and h represents the implicit features. The decoding module is used to reconstruct the original input features from the implicit features learned by the encoding module through the encoding process. This can be represented by a decoding function r = g(h), where r represents the reconstructed original input features, i.e., the multimodal reconstructed features. The multimodal reconstructed features r are similar to the multimodal illumination features x. The decoding process is the inverse of the encoding process. The encoding and decoding modules have symmetrical neural network structures. The encoding and decoding modules can be neural networks with only one hidden layer or multiple hidden layers. The neural network types can be fully connected layers, convolutional layers, etc., without limitation here.

[0121] Here, the multimodal illumination features are reconstructed through the encoding module and decoding module of the pre-trained virtual scene processing model, and the multimodal illumination features are reconstructed into multimodal reconstruction features, which facilitates the subsequent construction of reconstruction errors based on the multimodal illumination features and multimodal reconstruction features, and accurately judges whether the illumination is abnormal through the reconstruction error, thereby obtaining accurate illumination detection results.

[0122] In step S208 , the server determines a reconstruction error of the virtual scene to be processed based on the multimodal illumination features and the multimodal reconstruction features.

[0123] In an embodiment of the present application, determining the reconstruction error of the virtual scene to be processed based on the multimodal illumination features and the multimodal reconstruction features can be achieved in the following manner: first, obtaining the eigenvalues of each dimension in the multimodal illumination vector corresponding to the multimodal illumination features and the eigenvalues of each dimension in the multimodal reconstruction vector corresponding to the multimodal reconstruction features; then, the multimodal illumination vector and the multimodal reconstruction vector have the same dimension; finally, determining the difference between the eigenvalues of each dimension in the multimodal illumination vector and the eigenvalues of the corresponding dimension in the multimodal reconstruction vector, and determining the sum of the squares of the differences; and determining the reconstruction error of the virtual scene to be processed based on the sum of the squares.

[0124] That is to say, the multimodal illumination vector corresponding to the multimodal illumination feature and the multimodal reconstruction vector corresponding to the multimodal reconstruction feature have the same dimension. In the process of constructing the reconstruction error, first, the eigenvalue of each dimension in the multimodal illumination vector is subtracted from the eigenvalue of the corresponding dimension in the multimodal reconstruction vector to obtain the difference between the two in each dimension. Then, the difference in each dimension is squared, and the squared values of each dimension are added to obtain the sum of squares. Finally, the sum of squares is divided by the number of dimensions to obtain the reconstruction error of the virtual scene to be processed, and the reconstruction error is a mean. The reconstruction error is used to determine whether the illumination in the virtual scene to be processed is abnormal by measuring the error between the original input features (multimodal illumination features) of the pre-trained virtual scene processing model and the output features (multimodal reconstruction features) of the pre-trained virtual scene processing model. If the reconstruction error is within a reasonable error range, it means that the original input features of the virtual scene processing model are normal features, that is, the lighting in the virtual scene to be processed is normal; if the reconstruction error exceeds the reasonable error range, it means that the original input features of the virtual scene processing model are abnormal features, that is, the lighting in the virtual scene to be processed is abnormal.

[0125] Here, the reconstruction error is constructed based on the multimodal illumination features and the multimodal reconstruction features, so that the reconstruction error can be used to accurately judge whether the illumination in the virtual scene to be processed is abnormal and obtain accurate illumination detection results.

[0126] In step S209 , the server uses a detection threshold corresponding to the virtual scene to be processed to perform error analysis on the reconstruction error to obtain a lighting detection result of the virtual scene to be processed.

[0127] In an embodiment of the present application, the detection threshold corresponding to the virtual scene to be processed can be achieved by: obtaining multiple scene object data of the virtual scene to be processed and the scene weight corresponding to each scene object data; integrating the product of each scene object data and the scene weight respectively to obtain multiple integration results; obtaining the integration weight corresponding to each integration result; adding the product of the multiple integration results and the integration weight to obtain the detection threshold corresponding to the virtual scene to be processed. That is to say, first, the multiple scene object data of the virtual scene to be processed are weighted using the scene weight corresponding to each scene object data, then the weighted results are integrated respectively, and a nonlinear relationship is introduced through the integration processing process, and finally, the multiple integration results are weighted using the integration weight, and the weighted integration results are added to obtain the detection threshold corresponding to the virtual scene to be processed. For different virtual scenes to be processed, the multiple scene object data are independent variables, and the detection threshold corresponding to the virtual scene to be processed is the dependent variable.

[0128] Here, when the data of multiple scene objects in the virtual scene to be processed changes, the corresponding detection threshold will also change. That is, the detection threshold in the lighting detection process can be automatically and dynamically adjusted according to the simplicity or complexity of the virtual scene to be processed, so that the lighting detection process can adapt to the lighting changes of different scenes to be processed and has strong robustness.

[0129] In some embodiments, the illumination detection result can be determined as follows: if the reconstruction error is less than or equal to a detection threshold, the illumination detection result for the virtual scene to be processed is determined to be normal illumination; if the reconstruction error is greater than the detection threshold, the illumination detection result for the virtual scene to be processed is determined to be abnormal illumination. In other words, the reconstruction error is numerically compared with the detection threshold, and the comparison result can be used to determine whether the illumination detection result for the virtual scene to be processed is normal illumination or abnormal illumination.

[0130] In step S210 , the server encodes the lighting attribute features through the encoding module of the pre-trained virtual scene processing model to obtain encoded lighting attribute features.

[0131] In an embodiment of the present application, the encoding module of the pre-trained virtual scene processing model can jointly encode multimodal lighting features and lighting attribute features. The encoding module can better capture the lighting information in the virtual scene to be processed, which is conducive to the subsequent accurate lighting prediction results and reconstruction errors. The lighting attribute features correspond to different types of features related to the lighting attributes in the virtual scene to be processed. The different types of lighting attribute features are regarded as the inputs of multiple branch tasks in the lighting prediction process, and the outputs of the multiple branch tasks are subsequently obtained, namely the lighting prediction results after feature mapping.

[0132] In step S211 , the server performs feature mapping on the encoded illumination attribute features through a feature mapping module of a pre-trained virtual scene processing model to obtain an illumination prediction result for the virtual scene to be processed.

[0133] In an embodiment of the present application, a feature mapping module is used to perform feature mapping on the encoded lighting attribute features. The feature mapping module can be a single-layer neural network or a multi-layer neural network. The type of neural network included in the feature mapping module can be modified based on the actual scene requirements or the lighting prediction results of the virtual scene to be processed. The lighting prediction results output by the feature mapping module can help effectively locate whether there are any lighting anomalies in the virtual scene to be processed and determine the cause of any lighting anomalies.

[0134] In step S212 , when the illumination detection result indicates illumination anomaly, the server performs parameter anomaly analysis on the illumination prediction result to obtain illumination parameters to be adjusted in the virtual scene to be processed.

[0135] In the embodiment of the present application, the lighting prediction result refers to the lighting information in the virtual scene to be processed that is predicted through the lighting prediction process of the pre-trained virtual scene processing model. For example, the light source position, lighting intensity, lighting direction, lighting distance, etc. can be predicted. The parameter anomaly analysis is used to compare the lighting information in the lighting prediction result with the standard lighting parameters when the lighting detection result is a lighting anomaly, to determine whether the lighting information in the lighting prediction result is consistent with the standard lighting parameters, or to determine whether the difference between the lighting information in the lighting prediction result and the standard lighting parameters is within a reasonable error range, to obtain the parameter anomaly analysis result. When the lighting information in the lighting prediction result is inconsistent with the standard lighting parameters, or the difference between the lighting information in the lighting prediction result and the standard lighting parameters exceeds a reasonable error range, it is determined that the lighting information is the lighting parameter to be adjusted in the virtual scene to be processed. For example, the predicted light source position is compared with the standard light source position. If the predicted light source position is inconsistent with the standard light source position, or the difference between the predicted light source position and the standard light source position exceeds a reasonable error range, the light source position is determined as the lighting parameter to be adjusted in the virtual scene to be processed.

[0136] Here, the parameter anomaly analysis process is used to determine the lighting parameters to be adjusted in the virtual scene to be processed, so that the lighting anomaly problem of the virtual scene to be processed can be targeted and optimized in real time based on the lighting parameters to be adjusted.

[0137] Step S213: determining a lighting processing strategy for the virtual scene to be processed based on the lighting parameters to be adjusted.

[0138] In an embodiment of the present application, the lighting processing strategy is used to adjust the lighting parameters to be adjusted for the virtual scene to be processed, or to use advanced lighting technology to achieve real-time lighting optimization. For example, if the lighting parameter to be adjusted is light intensity, the lighting processing strategy is to adjust the light intensity of the virtual scene to be processed, or to use dynamic ambient light occlusion technology to achieve control of the light intensity. Here, through the lighting optimization process, it is helpful to achieve a more natural and realistic lighting effect in the virtual scene to be processed, and improve the rendering quality of the virtual scene to be processed.

[0139] Step S214: The server sends the lighting processing strategy to the terminal.

[0140] Step S215: The terminal outputs the lighting processing strategy.

[0141] In the embodiment of the present application, on the one hand, data preprocessing of multimodal image data and illumination attribute data is completed through normalization processing, feature extraction and feature fusion processes. The multimodal illumination features after data preprocessing are then used as input features of the illumination detection process. The multimodal illumination features contain feature information of multiple modalities in the virtual scene to be processed. Through the encoding module and decoding module of unsupervised learning, the output features of the illumination detection process, namely the multimodal reconstruction features, are reconstructed. And using the reconstruction error and detection threshold constructed based on the multimodal illumination features and the multimodal reconstruction features, accurate illumination detection results are obtained, thereby improving the accuracy of illumination detection. In addition, the detection threshold can be dynamically adjusted according to the complexity of the virtual scene to be processed, thereby improving the robustness of illumination detection.

[0142] On the other hand, the lighting attribute data is used as the input feature of the lighting prediction process, and the final lighting prediction result is obtained through the encoding module and feature mapping module of multi-task learning, so that the lighting prediction process is used as a branch task of virtual scene processing. When the lighting detection result is a lighting anomaly, the parameters to be adjusted in the virtual scene to be processed are determined based on the lighting prediction result, so as to accurately locate the cause of the lighting anomaly and quickly and accurately formulate a lighting processing strategy, so as to use the lighting processing strategy to achieve real-time lighting optimization of the virtual scene to be processed.

[0143] In some embodiments, the illumination detection results and illumination prediction results in the above virtual scene processing method can be realized by a pre-trained virtual scene processing model, see Figure 5 , Figure 5 The flowchart of the pre-trained virtual scene processing model training method provided by the embodiment of the present application is shown. The pre-trained virtual scene processing model training method of the embodiment of the present application can be executed by a model training module, wherein the model training module can be a module in an electronic device for implementing the virtual scene processing method, that is, the pre-trained virtual scene processing model training method can be executed by a terminal or a server; of course, the model training module can also be a module in other electronic devices different from the electronic device for implementing the virtual scene processing method, that is, the pre-trained virtual scene processing model training method can be executed by other terminals or other servers. Figure 5 As shown, the pre-trained virtual scene processing model is trained through the following steps S301 to S308:

[0144] Step S301: Obtain sample data.

[0145] In an embodiment of the present application, the sample data includes multimodal image sample data and lighting attribute sample data; the lighting attribute sample data is the lighting attribute information of at least one branch task associated with the lighting detection task. The multimodal image sample data is the input of the lighting detection task, and the lighting attribute sample data is the input of at least one branch task. The lighting detection task is used to detect the lighting of the virtual scene to be processed, and at least one branch task is used to assist in determining lighting anomaly parameters and accurately locate lighting anomaly problems. The number of branch tasks can be adjusted according to the specific lighting information in the virtual scene to be processed and is not limited here.

[0146] In step S302 , data preprocessing is performed on the multimodal image sample data and the illumination attribute sample data respectively to obtain multimodal illumination sample features and illumination attribute sample features, and the multimodal illumination sample features and illumination attribute sample features are input into the virtual scene processing model.

[0147] In an embodiment of the present application, the virtual scene processing model is a network model of a pre-trained virtual scene processing model before model training. That is, after model training, the virtual scene processing model obtains a pre-trained virtual scene processing model, and the virtual scene processing model and the pre-trained virtual scene processing model have the same model structure. The virtual scene processing model includes an encoding module, a decoding module, and a feature mapping module. The multimodal illumination sample features and illumination attribute sample features after data preprocessing serve as the model input of the virtual scene processing model.

[0148] In an embodiment of the present application, the specific process of performing data preprocessing on the multimodal image sample data and the illumination attribute sample data is as follows: the multimodal image sample data and the illumination attribute sample data of the virtual scene to be processed are normalized respectively, and normalized multimodal image sample data and normalized illumination attribute sample data are obtained accordingly; feature extraction is performed on the normalized multimodal image sample data and the normalized illumination attribute sample data respectively, and multimodal sample feature vectors and illumination attribute sample features are obtained accordingly; feature fusion is performed on the multimodal sample feature vectors to obtain multimodal illumination sample features.

[0149] Step S303 : encoding the multimodal illumination sample features and the illumination attribute sample features respectively through the encoding module of the virtual scene processing model to obtain encoded multimodal illumination sample features and encoded illumination attribute sample features.

[0150] In an embodiment of the present application, the multimodal illumination sample features and the illumination attribute sample features are jointly input into the encoding module of the virtual scene processing model, and the encoding module is used to encode the multimodal illumination sample features and the illumination attribute sample features, so that the encoding module can better capture the illumination information in the virtual scene to be processed, which is conducive to the subsequent accurate illumination sample prediction results and sample reconstruction errors.

[0151] In step S304, the encoded multimodal illumination sample features are decoded by the decoding module of the virtual scene processing model to obtain multimodal reconstruction sample features, and the encoded illumination attribute sample features are feature mapped by the feature mapping module of the virtual scene processing model to obtain illumination sample prediction results.

[0152] In an embodiment of the present application, the decoding module of the virtual scene processing model is used to decode the encoded multimodal illumination sample features and obtain multimodal reconstruction sample features after decoding. The multimodal reconstruction sample features can be used to construct the sample reconstruction error of the subsequent illumination detection task. The feature mapping module is used to perform feature mapping on the encoded illumination attribute sample features and obtain the illumination sample prediction results for each branch task after feature mapping. The illumination sample prediction results can be used to construct the branch loss of each subsequent branch task.

[0153] Step S305 : constructing a sample reconstruction error of the illumination detection task based on the multimodal illumination sample features and the multimodal reconstruction sample features.

[0154] In an embodiment of the present application, the sample reconstruction error of the lighting detection task is the difference between the multimodal reconstruction sample features and the multimodal lighting sample features. The sample reconstruction error is used to evaluate whether the lighting in the virtual scene to be processed is abnormal. During the model training process, it can be regarded as part of the model loss, and the model parameters of the virtual scene processing model can be updated based on the sample reconstruction error.

[0155] Step S306 : determining the branch loss of each branch task based on the illumination sample prediction result and the label of the illumination attribute sample data.

[0156] In the embodiment of the present application, the illumination sample prediction result includes the illumination sample prediction result of each branch task, and the label of the illumination attribute sample data includes the label of the illumination attribute sample data of each branch task. Based on the illumination sample prediction result and the label of the illumination attribute sample data of each branch task, a branch loss for each branch task is constructed. For example, the branch loss can be a cross-entropy loss function.

[0157] Step S307: determining the loss result of the virtual scene processing model based on the sample reconstruction error and the branch loss.

[0158] In an embodiment of the present application, determining the loss result of the virtual scene processing model based on the sample reconstruction error and branch loss can be achieved in the following manner: obtaining the weight of each branch loss in at least one branch loss; based on the weight, performing weighted processing on each branch loss to obtain the weighted loss of each branch task; based on the reconstruction error and the weighted loss, constructing the model loss of the virtual scene processing model; performing loss calculation on the model loss to obtain the loss result of the virtual scene processing model.

[0159] That is to say, the model loss of the virtual scene processing model is composed of the reconstruction error and the weighted loss of each branch task. The model loss is calculated to obtain the loss result of the virtual scene processing model. The loss result is used to measure the inconsistency between the multimodal illumination sample features and multimodal reconstruction sample features, the illumination sample prediction results and the labels of the illumination attribute sample data of the virtual scene processing model, that is, the gap between the forward calculation results of each iteration of the calculation model (that is, the multimodal reconstruction sample features and the illumination sample prediction results) and the true value (that is, the labels of the multimodal reconstruction sample features and the illumination attribute sample data), thereby guiding the next step of training in the right direction.

[0160] Step S308: Update the model parameters of the virtual scene processing model based on the loss result to obtain a pre-trained virtual scene processing model.

[0161] In an embodiment of the present application, the model parameters of the virtual scene processing model are updated based on the loss result to obtain a pre-trained virtual scene processing model. This can be achieved by: determining the parameter gradient of the model parameters of the virtual scene processing model; determining the model deletion parameters of the virtual scene processing model based on the parameter gradient and a preset gradient threshold; performing parameter deletion on the model parameters based on the model deletion parameters to obtain the model update parameters of the virtual scene processing model; and updating the model update parameters of the virtual scene processing model based on the loss result to obtain the pre-trained virtual scene processing model. That is, during the model training process of the virtual scene processing model, the parameter gradient of the model parameters of each layer of the neural network of the virtual scene processing model is output, and the parameter gradient is numerically compared with the preset gradient threshold. If the parameter gradient is less than the preset gradient threshold, it means that the model parameter corresponding to the parameter gradient is a non-critical parameter of the virtual scene processing model, and the model parameter is determined as a model deletion parameter; if the parameter gradient is greater than or equal to the preset gradient threshold, it means that the model parameter corresponding to the parameter gradient is a critical parameter of the virtual scene processing model. The model deletion parameter does not participate in the model update process of the virtual scene processing model, and the model update parameter after parameter deletion is updated. Based on the derivative of the model loss, the loss result is propagated back along the direction of minimum gradient to update the model update parameters in the encoding module, decoding module, and feature mapping module of the virtual scene processing model, such as the weight values in the encoding module, decoding module, and feature mapping module. A loss result threshold is pre-set. When the loss result is less than the pre-set loss result threshold, iterative training is stopped, i.e., model parameter updates are stopped. A maximum iteration number threshold can also be pre-set. When the number of iterations exceeds the maximum iteration number threshold, model parameter updates are stopped. A cutoff iteration time can also be pre-set. When the iteration time reaches the cutoff iteration time, model parameter updates are stopped, resulting in a pre-trained virtual scene processing model. The model training process of the pre-trained virtual scene processing model is an offline training process. The pre-trained virtual scene processing model is stored offline after training. The pre-trained virtual scene processing model can be used during virtual scene processing to perform real-time detection and prediction of lighting anomalies in the virtual scene to be processed.

[0162] Here, the process of determining the model update parameters is a process of optimizing and accelerating the virtual scene processing model. This reduces the model size and computational complexity of the virtual scene processing model, thereby improving the model's real-time detection performance. In other embodiments, model optimization and acceleration can also be achieved through methods such as quantization and knowledge distillation.

[0163] In an embodiment of the present application, the multimodal illumination sample features and illumination attribute sample features that have undergone data preprocessing are input into a virtual scene processing model. After passing through the encoding module, decoding module, and feature mapping module in the virtual scene processing model, the decoded multimodal reconstruction sample features and the illumination sample prediction results after feature mapping are obtained. Based on the multimodal reconstruction sample features and the multimodal illumination sample features, the loss between the illumination sample prediction results and the labels is calculated, and model pruning is performed during the training process to reduce the model training parameters. The optimal pre-trained virtual scene processing model is trained to obtain the optimal sample reconstruction error and illumination sample prediction results, and then based on the pre-trained virtual scene processing model, effective, accurate, and real-time illumination detection is achieved for the virtual scene to be processed based on the sample reconstruction error and illumination sample prediction results, thereby improving the visual quality and realism of the virtual scene to be processed.

[0164] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0165] The present invention provides a method for processing virtual scenes. This method involves a multimodal intelligent lighting detection method. This method utilizes real-time lighting anomaly detection technology based on multimodal input and innovative unsupervised learning methods, aiming to provide game developers with an efficient, accurate, and real-time lighting anomaly detection solution. From a product perspective, this method has the following key features: Easy Integration: This method can be easily integrated into existing game engines (such as UE4) to provide real-time lighting anomaly detection capabilities. By creating a game engine plug-in, developers can seamlessly utilize the lighting anomaly detection service provided by this method during the game development process. User-Friendliness: The method's real-time lighting analysis capabilities can help game developers more quickly identify potential lighting issues, improving the visual quality and realism of their games. Furthermore, model interpretability research can provide developers with targeted optimization suggestions, helping them more efficiently resolve lighting anomaly issues. High Performance: This method utilizes model optimization and acceleration technologies to improve the real-time performance of the model within the game engine, ensuring smooth game operation. This enables the present invention to provide high-performance real-time lighting anomaly detection capabilities in gaming scenarios. Powerful generalization performance: This method designs an adaptive threshold and parameter adjustment method, which enables the model to automatically adjust the threshold and parameters of lighting anomaly detection according to the characteristics of the scene, and can achieve efficient lighting anomaly detection in a variety of game scenes and lighting conditions. Reduce development costs: This method adopts an unsupervised learning method, which reduces the dependence on a large amount of labeled lighting anomaly data, thereby reducing development costs. At the same time, through real-time lighting anomaly detection, developers can discover and solve lighting problems more quickly, further reducing game development costs. In summary, the method provided in the embodiment of the present application provides game developers with a real-time, efficient, accurate and easy-to-integrate lighting anomaly detection technology from the product side, which helps to improve the visual quality and realism of the game, reduce game development costs, and improve development efficiency.

[0166] The framework diagram of the multimodal lighting intelligent detection provided by the embodiment of the present application is as follows: Figure 6 As shown, it mainly includes multimodal data input, preprocessing and fusion, unsupervised learning model training, real-time lighting anomaly detection, adaptive threshold and parameter adjustment, optimization suggestion generation and game lighting optimization.

[0167] The multimodal input processing module is a key component of real-time illumination anomaly detection technology. It is responsible for processing multimodal data from the scene, including RGB images, depth maps, and normal maps. By preprocessing and fusing this data, it can fully utilize the scene's color, geometry, and illumination information, providing rich input features for subsequent illumination anomaly detection.

[0168] First, multimodal scene data (i.e., the aforementioned multimodal image data) is collected in real time from the game engine or rendering system. The RGB image (i.e., the aforementioned RGB image data) contains the scene's color information, the depth map (i.e., the aforementioned depth image data) reflects the distance relationships between objects in the scene, and the normal map (i.e., the aforementioned normal image data) describes the surface normals of the objects. Together, these data provide rich input features for illumination anomaly detection (compared to traditional illumination anomaly detection techniques based on RGB images).

[0169] In a game engine (e.g., Unity or UE4), you can write scripts or plugins to capture multimodal data of the scene in real time. For example, in Unity, you can use the Camera.RenderWithShader method to render a depth map and a normal map, and use RenderTexture to save the rendered results. Figure 7 As shown, the depth map of the scene is collected in real time. Figure 8 As shown, the normal map of the scene is collected in real time. Figure 9 As shown in Figure 3, depth maps and normal maps contain more feature information compared to RGB images.

[0170] / / Sample code for collecting depth map and normal map in Unity:

[0171] Camera cam=GetComponent <camera>();

[0172] RenderTexture depthTexture=new RenderTexture(width,height,0);

[0173] RenderTexture normalTexture=new RenderTexture(width,height,0);

[0174] Shader depthShader=Shader.Find("Custom / DepthShader");

[0175] Shader normalShader=Shader.Find("Custom / NormalShader");

[0176] cam.RenderWithShader(depthShader,"",depthTexture);

[0177] cam.RenderWithShader(normalShader,"",normalTexture);

[0178] / / Output depth map and normal map;

[0179] For each input image (RGB image, depth map, and normal map), it needs to be preprocessed to match the input requirements of the pre-trained model. Preprocessing includes scaling the image to a fixed size (such as 224x224) and normalizing the image. Let the input image be I(x,y), the preprocessed image be I′(x,y), and the normalization parameters be the mean μ and standard deviation σ of the input image. The preprocessing process can be expressed as:

[0180] I′(x,y)=(I(x,y)-μ) / σ (1)

[0181] For each preprocessed image (I'_rgb, I'_depth, and I'_normal), a pre-trained convolutional neural network (e.g., VGG-16) is used to extract features. The network structure and network parameters of the pre-trained convolutional neural network are as follows: Figure 10 As shown. Assuming the feature extraction function is F, the extracted features are:

[0182]

[0183] The extracted RGB image features F_rgb, depth map features F_depth, and normal map features F_normal are fused to form a unified feature representation F_fused. Assuming that feature concatenation is used as the fusion method, the fused feature F_fused (i.e., the multimodal illumination feature mentioned above) is expressed as:

[0184] F_fused=[F_rgb,F_depth,F_normal] (3)

[0185] Among them, [,,] represents the feature cascade operation.

[0186] Through feature fusion operation, multi-dimensional features are reduced to two dimensions and represented by scatter plots. Figure 11 shown.

[0187] After obtaining the fused feature F_fused, it can be used as input data for subsequent model training. According to the task requirements, a suitable model can be selected for training. The fused feature F_fused is a cascade of feature vectors extracted from the RGB image, depth map, and normal map. In the previous example, a pre-trained VGG-16 model was used to extract features. The VGG-16 model outputs a feature vector of size 4096 before the last fully connected layer. Therefore, when features are extracted from the RGB image, depth map, and normal map and concatenated, a fused feature of size 3*4096=12288 can be obtained. The output of the fused feature is as follows Figure 12 shown.

[0188] For the unsupervised model training part of the autoencoder, the autoencoder is an unsupervised learning method used to learn an effective representation of the input data. It consists of two parts: an encoder (i.e. the encoding module mentioned above) and a decoder (i.e. the decoding module mentioned above). The encoder compresses the input data into a low-dimensional representation, and the decoder restores the low-dimensional representation to the original input data. By training the autoencoder to minimize the difference between the input data and the restored data, a compact representation of the input data can be obtained. Due to the high amount and cost of labeled data, the autoencoder unsupervised method is chosen for model training. The encoding neural network structure is as follows Figure 13 As shown, the decoding neural network structure is as follows Figure 14 As shown in the figure, the encoding and decoding blocks form an unsupervised learning network structure.

[0189] Regarding innovations in the autoencoder structure, an attempt was made to design an autoencoder structure specific to the illumination anomaly detection task. A convolutional autoencoder can be used to better capture the local structure of image features. The illumination anomaly detection strategy is based on reconstruction error, where the reconstruction error is the error between the input of the convolutional autoencoder and the output of the decoder. In a convolutional autoencoder, the reconstruction error of normal data is typically small, while the reconstruction error of abnormal data is large. Therefore, the reconstruction error of the test data can be calculated and compared with a pre-set reconstruction error threshold (i.e., the detection threshold mentioned above). When the reconstruction error exceeds the reconstruction error threshold, the data is considered anomaly. To determine the appropriate reconstruction error threshold, a validation dataset can be used to evaluate the detection performance (such as accuracy, recall, and F1 score) under different reconstruction error thresholds, and the reconstruction error threshold that achieves the best performance can be selected.

[0190] The reconstruction error can be expressed as the mean square error (MSE), which is the average of the squares of the differences between the original data point x (i.e., the multimodal illumination feature) and the reconstructed data point x′ (i.e., the multimodal reconstruction feature). For n-dimensional data points, the reconstruction error is calculated as follows:

[0191]

[0192] Among them, x i and x i ′ represents the i-th component of the original data point x and the reconstructed data point x′, ∑() is the accumulation function, and Loss reconstruction Represents the reconstruction error. In this way, the pre-trained model is obtained through training. Since the reconstruction error of abnormal data is large, the reconstruction error of input and output can be used to determine whether the test data is abnormal. The loss curve of unsupervised learning based on autoencoder is as follows Figure 15 shown.

[0193] The above model is a basic model, but in order to improve the generalization ability of the model and adapt to different scene lighting changes, multi-task learning is also introduced in this solution. First, some branch tasks related to lighting need to be determined. These branch tasks can help the autoencoder better capture the lighting information in the scene. For example: predicting the position of the light source, light intensity, light direction, etc. On the basis of the autoencoder, a separate output branch can be added for each branch task. For example, if the branch task is to predict the position of the light source, a fully connected layer can be added after the output of the encoder to predict the position coordinates. The multi-task network structure diagram of the autoencoder is as follows: Figure 16 As shown in Figure 2, the output of the fully connected layer can be used for a variety of branch tasks, such as classification, regression, and feature extraction. In specific applications, the n branch task outputs of the fully connected layer can be adjusted according to task requirements. For example, for classification tasks, the softmax function can be used to convert the output into a probability distribution. Here, the branch tasks can output the direction and intensity of the lighting, which can better help locate any lighting anomalies.

[0194] Define the multi-task loss function (i.e., the model loss mentioned above): In order to train a multi-task network, it is necessary to define a multi-task loss function that considers both the reconstruction error and the error of the branch task (i.e., the branch loss mentioned above). The loss function can be defined as the weighted sum of the losses of each task, as shown in formula (5):

[0195] Loss total =Loss reconstruction +λ1*Loss task1 +λ2*Loss task2 +…+λ n *Loss taskn (5)

[0196] Among them, Loss total is the total loss function, Loss reconstruction is the reconstruction error, Loss task1 、Loss task2 ,…,Loss taskn is the loss function corresponding to n branch tasks, λ1, λ2, ..., λ n is the weight of the loss function corresponding to the n branch tasks. In this way, the trained model will have stronger generalization ability and can take into account more environmental information such as light intensity and position.

[0197] In order to improve the real-time performance of the model in the game engine and ensure that the game runs smoothly, the present invention adopts model optimization and acceleration technology, such as model pruning technology, during the model training process. Pruning is performed based on the importance of model parameters. Since too many model parameters will lead to poor model performance, the importance of each parameter in the model can be evaluated after the model training is completed. The evaluation can be achieved through a variety of methods, such as calculating the gradient and second-order derivative (Hessian) of the parameters. In the autoencoder, the importance of the parameters in the convolutional layers of the encoder and decoder can be evaluated. By calculating the parameter gradient, the weight of the influence of the parameter on the model result can be obtained. Output the parameter gradient. If the parameter has little effect on the model result, the gradient value of the parameter is small, then the parameter is directly turned off to reduce non-critical parameters and obtain a new model. The specific process of implementing the model pruning technology is as follows: Figure 17 shown.

[0198] In addition, in order to achieve real-time illumination detection, a dynamic reconstruction error threshold can be used to determine whether the reconstruction error is abnormal. For example, the reconstruction error threshold can be set to a certain percentile (for example, the 95th percentile) of the reconstruction error of the training set, and the reconstruction error threshold can also be calculated based on the number of scene objects in the scene information (for example, the magnitude of trees, flowers and plants). In this way, when the scene becomes more complex or simpler, the reconstruction error threshold will rise or fall accordingly, making anomaly detection more robust. Based on the comparison result of the calculated reconstruction error and the dynamic reconstruction error threshold, it can be determined whether the illumination of the current scene is abnormal, that is, the detection result is output. The logic diagram for dynamic adjustment of the reconstruction error threshold is as follows: Figure 18 As shown, the dynamic reconstruction error threshold calculation formula is shown in formula (6).

[0199] T=k1*∫(a*N(t)dt)+k2*∫(b*O(t)dt)+k3*∫(c*R(t)dt)+d (6)

[0200] Where T represents the reconstruction error threshold, N(t) represents the number of triangles at time t; O(t) represents the number of scene objects at time t; R(t) represents the texture load at time t; a, b, and c are the weights of the number of triangles, the number of scene objects, and the texture load, respectively; k1, k2, k3, and d are adjustable parameters, representing the weight and intercept after integration, respectively; ∫ represents the integral sign, which is used to introduce a nonlinear relationship. This formula introduces a nonlinear relationship by integrating the number of triangles, the number of scene objects, and the texture load, taking into account the number of objects, the number of triangles (indicating whether the object rendering is complex), and the threshold for texture load calculation.

[0201] Real-time lighting optimization: If lighting anomalies are detected, measures can be taken to optimize the lighting. This can include adjusting parameters such as the position, direction, and intensity of the light source, or using advanced techniques such as image-based lighting and dynamic ambient occlusion. When performing lighting optimization, it is important to ensure that the operation is as fast as possible to meet real-time requirements.

[0202] The method provided in the embodiment of the present application can be applied to the routine lighting anomaly detection test of the image source management tool (NPM Registry Manager, NZM) and perform continuous daily iterations. The above-mentioned real-time lighting detection and optimization scheme based on the unsupervised learning autoencoder model has the following benefits and effects: 1. Automatic detection of lighting anomalies: By calculating the reconstruction error and comparing it with the dynamic threshold, the method can automatically detect lighting anomalies in real-time scenes. This helps to quickly discover rendering errors, unnatural lighting or other lighting problems, thereby improving the rendering quality. 2. Real-time performance: The method can achieve real-time lighting detection and optimization under GPU optimization. During the rendering process, lighting anomaly detection and optimization can be carried out synchronously with scene updates, thereby achieving high-quality real-time rendering effects. 3. Generalization performance: By using dynamic thresholds, multi-task learning, combining depth information, using timing information and multimodal fusion, the method can have good generalization performance in different scenarios. When dealing with various complex scenes, the lighting anomaly detection and optimization effects remain stable and reliable. 4. Adaptive lighting optimization: When lighting anomalies are detected, this method can automatically optimize the lighting based on the characteristics of the scene. This may include adjusting light source parameters, using advanced lighting techniques, etc., which helps to achieve more natural and realistic lighting effects and improve rendering quality.

[0203] It can be understood that in the embodiments of the present application, the content involving user information, such as multimodal image data, lighting attribute data, lighting processing strategies and other information, if it involves data related to user information or corporate information, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0204] The following continues to describe the exemplary structure of the virtual scene processing device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the virtual scene processing device 455 of the memory 450 may include: a data preprocessing module 4551, which is used to perform data preprocessing on the multimodal image data and illumination attribute data of the virtual scene to be processed, respectively, to obtain multimodal illumination features and illumination attribute features correspondingly; a reconstruction error determination module 4552, which is used to determine the reconstruction error of the virtual scene to be processed based on the multimodal illumination features; an error analysis module 4553, which is used to use a detection threshold corresponding to the virtual scene to be processed to perform error analysis on the reconstruction error to obtain the illumination detection result of the virtual scene to be processed; an illumination prediction result determination module 4554, which is used to determine the illumination prediction result of the virtual scene to be processed based on the illumination attribute features; and an illumination processing strategy determination module 4555, which is used to determine the illumination processing strategy of the virtual scene to be processed based on the illumination detection result and the illumination prediction result.

[0205] In some embodiments, the reconstruction error determination module 4552 is further used to: perform feature reconstruction on the multimodal illumination feature to obtain a multimodal reconstruction feature; and determine the reconstruction error of the virtual scene to be processed based on the multimodal illumination feature and the multimodal reconstruction feature.

[0206] In some embodiments, the reconstruction error determination module 4552 is also used to: encode the multimodal illumination features through the encoding module of the pre-trained virtual scene processing model to obtain the encoded multimodal illumination features; and decode the encoded multimodal illumination features through the decoding module of the pre-trained virtual scene processing model to obtain the multimodal reconstruction features.

[0207] In some embodiments, the reconstruction error determination module 4552 is also used to: obtain the eigenvalues of each dimension in the multimodal illumination vector corresponding to the multimodal illumination feature and the eigenvalues of each dimension in the multimodal reconstruction vector corresponding to the multimodal reconstruction feature; the multimodal illumination vector has the same dimension as the multimodal reconstruction vector; determine the difference between the eigenvalues of each dimension in the multimodal illumination vector and the eigenvalues of the corresponding dimension in the multimodal reconstruction vector, and determine the sum of the squares of the differences; and determine the reconstruction error of the virtual scene to be processed based on the sum of the squares.

[0208] In some embodiments, the device 455 also includes a detection threshold determination module, and the detection threshold determination is used to: obtain multiple scene object data of the virtual scene to be processed and the scene weight corresponding to each of the scene object data; integrate the products of each of the scene object data and the scene weight respectively to obtain multiple integration results; obtain the integration weight corresponding to each of the integration results; add the products of the multiple integration results and the integration weight to obtain the detection threshold corresponding to the virtual scene to be processed.

[0209] In some embodiments, the error analysis module 4553 is also used to: when the reconstruction error is less than or equal to the detection threshold, determine that the lighting detection result of the virtual scene to be processed is normal lighting; when the reconstruction error is greater than the detection threshold, determine that the lighting detection result of the virtual scene to be processed is abnormal lighting.

[0210] In some embodiments, the lighting prediction result determination module 4554 is also used to: encode the lighting attribute features through the encoding module of the pre-trained virtual scene processing model to obtain the encoded lighting attribute features; and perform feature mapping on the encoded lighting attribute features through the feature mapping module of the pre-trained virtual scene processing model to obtain the lighting prediction results of the virtual scene to be processed.

[0211] In some embodiments, the data preprocessing module 4551 is also used to: perform normalization processing on the multimodal image data and the illumination attribute data respectively, and obtain normalized multimodal image data and normalized illumination attribute data accordingly; perform feature extraction on the normalized multimodal image data and the normalized illumination attribute data respectively, and obtain multimodal feature vectors and the illumination attribute features accordingly; perform feature fusion on the multimodal feature vectors to obtain the multimodal illumination features.

[0212] In some embodiments, the lighting processing strategy determination module 4555 is also used to: when the lighting detection result is a lighting abnormality, perform parameter abnormality analysis on the lighting prediction result to obtain the lighting parameters to be adjusted under the virtual scene to be processed; based on the lighting parameters to be adjusted, determine the lighting processing strategy for the virtual scene to be processed.

[0213] In some embodiments, the illumination detection result and the illumination prediction result are realized by a pre-trained virtual scene processing model; the device 455 also includes a model training module, and the model training module is used to: obtain sample data, the sample data including multimodal image sample data and illumination attribute sample data; the illumination attribute sample data is the illumination attribute information of at least one branch task associated with the illumination detection task; the multimodal image sample data and the illumination attribute sample data are respectively preprocessed to obtain multimodal illumination sample features and illumination attribute sample features, and the multimodal illumination sample features and the illumination attribute sample features are input into the virtual scene processing model; the multimodal illumination sample features and the illumination attribute sample features are respectively encoded by the encoding module of the virtual scene processing model to obtain the encoded multimodal illumination sample features. The method comprises the following steps: decoding the encoded multimodal illumination sample features and the encoded illumination attribute sample features through the decoding module of the virtual scene processing model to obtain multimodal reconstruction sample features, and feature mapping the encoded illumination attribute sample features through the feature mapping module of the virtual scene processing model to obtain illumination sample prediction results; constructing the sample reconstruction error of the illumination detection task based on the multimodal illumination sample features and the multimodal reconstruction sample features; determining the branch loss of each branch task based on the illumination sample prediction results and the labels of the illumination attribute sample data; determining the loss result of the virtual scene processing model based on the sample reconstruction error and the branch loss; and updating the model parameters of the virtual scene processing model based on the loss result to obtain the pre-trained virtual scene processing model.

[0214] In some embodiments, the model training module is also used to: obtain the weight of each branch loss in the at least one branch loss; based on the weight, perform weighted processing on each branch loss to obtain the weighted loss of each branch task; based on the reconstruction error and the weighted loss, construct the model loss of the virtual scene processing model; perform loss calculation on the model loss to obtain the loss result of the virtual scene processing model.

[0215] In some embodiments, the model training module is also used to: determine the parameter gradient of the model parameters of the virtual scene processing model; determine the model deletion parameters of the virtual scene processing model based on the parameter gradient and a preset gradient threshold; perform parameter deletion on the model parameters based on the model deletion parameters to obtain the model update parameters of the virtual scene processing model; update the model update parameters of the virtual scene processing model based on the loss result to obtain the pre-trained virtual scene processing model.

[0216] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0217] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the virtual scene processing method provided by the embodiment of the present application, for example, Figure 3 The virtual scene processing method shown.

[0218] The present invention provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the virtual scene processing method described in the present invention.

[0219] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0220] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0221] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0222] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0223] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.< / camera>

Claims

1. A virtual scene processing method, characterized in that: The method comprises: Preprocess the multimodal image data and illumination attribute data of the virtual scene to be processed respectively, and obtain multimodal illumination features and illumination attribute features accordingly; Determining a reconstruction error of the virtual scene to be processed based on the multimodal illumination feature; Performing an error analysis on the reconstruction error using a detection threshold corresponding to the virtual scene to be processed, to obtain a lighting detection result of the virtual scene to be processed; Determining a lighting prediction result of the virtual scene to be processed based on the lighting attribute characteristics; Based on the illumination detection result and the illumination prediction result, an illumination processing strategy for the virtual scene to be processed is determined.

2. The method according to claim 1, characterized in that The determining, based on the multimodal illumination feature, a reconstruction error of the virtual scene to be processed, includes: Performing feature reconstruction on the multimodal illumination feature to obtain a multimodal reconstruction feature; A reconstruction error of the virtual scene to be processed is determined based on the multimodal illumination feature and the multimodal reconstruction feature.

3. The method according to claim 2, characterized in that The reconstructing the multimodal illumination feature to obtain a multimodal reconstructed feature includes: encoding the multimodal illumination features using an encoding module of a pre-trained virtual scene processing model to obtain encoded multimodal illumination features; The encoded multimodal illumination features are decoded by the decoding module of the pre-trained virtual scene processing model to obtain the multimodal reconstruction features.

4. The method according to claim 2, characterized in that The determining of the reconstruction error of the virtual scene to be processed based on the multimodal illumination feature and the multimodal reconstruction feature includes: Obtaining an eigenvalue in each dimension of a multimodal illumination vector corresponding to the multimodal illumination feature and an eigenvalue in each dimension of a multimodal reconstruction vector corresponding to the multimodal reconstruction feature; the multimodal illumination vector and the multimodal reconstruction vector have the same dimension; Determining a difference between an eigenvalue in each dimension of the multimodal illumination vector and an eigenvalue in the corresponding dimension of the multimodal reconstruction vector, and determining a sum of squares of the differences; A reconstruction error of the virtual scene to be processed is determined based on the sum of squares.

5. The method according to claim 1, wherein The method further comprises: Acquire a plurality of scene object data of the virtual scene to be processed and a scene weight corresponding to each of the scene object data; performing integration processing on the product of each of the scene object data and the scene weight respectively to obtain a plurality of integration results; Obtaining the integral weight corresponding to each integral result; The product of the multiple integration results and the integration weight is added to obtain a detection threshold corresponding to the virtual scene to be processed.

6. The method according to claim 5, characterized in that The step of adopting a detection threshold corresponding to the virtual scene to be processed to perform error analysis on the reconstruction error to obtain a lighting detection result of the virtual scene to be processed includes: When the reconstruction error is less than or equal to the detection threshold, determining that the illumination detection result of the virtual scene to be processed is normal illumination; When the reconstruction error is greater than the detection threshold, it is determined that the illumination detection result of the virtual scene to be processed is an illumination anomaly.

7. The method according to claim 1, characterized in that The determining, based on the illumination attribute characteristics, an illumination prediction result of the virtual scene to be processed includes: encoding the illumination attribute features through an encoding module of a pre-trained virtual scene processing model to obtain encoded illumination attribute features; The encoded illumination attribute features are feature mapped by the feature mapping module of the pre-trained virtual scene processing model to obtain an illumination prediction result of the virtual scene to be processed.

8. The method according to claim 1, characterized in that The multimodal image data and illumination attribute data of the virtual scene to be processed are respectively preprocessed to obtain multimodal illumination features and illumination attribute features, including: Normalizing the multimodal image data and the illumination attribute data respectively to obtain normalized multimodal image data and normalized illumination attribute data; Performing feature extraction on the normalized multimodal image data and the normalized illumination attribute data, respectively, to obtain a multimodal feature vector and the illumination attribute feature correspondingly; Feature fusion is performed on the multimodal feature vector to obtain the multimodal illumination feature.

9. The method according to claim 1, characterized in that The determining of the lighting processing strategy for the virtual scene to be processed based on the lighting detection result and the lighting prediction result includes: When the illumination detection result is illumination abnormality, performing parameter abnormality analysis on the illumination prediction result to obtain illumination parameters to be adjusted in the virtual scene to be processed; Based on the illumination parameters to be adjusted, a lighting processing strategy for the virtual scene to be processed is determined.

10. The method according to any one of claims 1 to 9, characterized in that The illumination detection result and the illumination prediction result are realized by a pre-trained virtual scene processing model; the pre-trained virtual scene processing model is trained by the following steps: Acquire sample data, where the sample data includes multimodal image sample data and illumination attribute sample data; the illumination attribute sample data is illumination attribute information of at least one branch task associated with the illumination detection task; performing data preprocessing on the multimodal image sample data and the illumination attribute sample data respectively to obtain multimodal illumination sample features and illumination attribute sample features respectively, and inputting the multimodal illumination sample features and the illumination attribute sample features into a virtual scene processing model; The encoding module of the virtual scene processing model encodes the multimodal illumination sample features and the illumination attribute sample features respectively to obtain encoded multimodal illumination sample features and encoded illumination attribute sample features; The encoded multimodal illumination sample features are decoded by the decoding module of the virtual scene processing model to obtain multimodal reconstruction sample features, and the encoded illumination attribute sample features are feature mapped by the feature mapping module of the virtual scene processing model to obtain illumination sample prediction results; Constructing a sample reconstruction error of the illumination detection task based on the multimodal illumination sample features and the multimodal reconstruction sample features; Determining a branch loss for each of the branch tasks based on the illumination sample prediction result and the label of the illumination attribute sample data; Determining a loss result of the virtual scene processing model based on the sample reconstruction error and the branch loss; The model parameters of the virtual scene processing model are updated based on the loss result to obtain the pre-trained virtual scene processing model.

11. The method according to claim 10, characterized in that The determining the loss result of the virtual scene processing model based on the sample reconstruction error and the branch loss includes: Obtaining a weight of each branch loss in the at least one branch loss; Based on the weight, weighting the loss of each branch is performed to obtain a weighted loss of each branch task; Constructing a model loss of the virtual scene processing model based on the reconstruction error and the weighted loss; The loss of the model is calculated to obtain a loss result of the virtual scene processing model.

12. The method according to claim 10, characterized in that The updating of the model parameters of the virtual scene processing model based on the loss result to obtain the pre-trained virtual scene processing model includes: determining parameter gradients of model parameters of the virtual scene processing model; Determining a model deletion parameter of the virtual scene processing model based on the parameter gradient and a preset gradient threshold; Based on the model deletion parameters, the model parameters are deleted to obtain model update parameters of the virtual scene processing model; Based on the loss result, the model update parameters of the virtual scene processing model are updated to obtain the pre-trained virtual scene processing model.

13. A virtual scene processing device, characterized in that: The device comprises: A data preprocessing module is used to preprocess the multimodal image data and illumination attribute data of the virtual scene to be processed, and obtain multimodal illumination features and illumination attribute features respectively; a reconstruction error determination module, configured to determine a reconstruction error of the virtual scene to be processed based on the multimodal illumination feature; an error analysis module, configured to perform error analysis on the reconstruction error using a detection threshold corresponding to the virtual scene to be processed, and obtain a lighting detection result of the virtual scene to be processed; An illumination prediction result determination module, configured to determine an illumination prediction result of the virtual scene to be processed based on the illumination attribute characteristics; The lighting processing strategy determination module is used to determine the lighting processing strategy of the virtual scene to be processed based on the lighting detection result and the lighting prediction result.

14. An electronic device, characterized in that: include: a memory for storing computer-executable instructions; The processor is configured to implement the virtual scene processing method according to any one of claims 1 to 12 when executing the computer executable instructions stored in the memory.

15. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the computer executable instructions are executed by a processor, the virtual scene processing method according to any one of claims 1 to 12 is implemented.

16. A computer program product, characterized in that The computer program product includes computer-executable instructions stored in a computer-readable storage medium; Wherein, when the processor of the electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, the virtual scene processing method according to any one of claims 1 to 12 is implemented.

Images