Rendering data-based scene recognition method and electronic equipment

Abstract

The invention provides a scene recognition method based on rendering data and electronic equipment, and the method comprises the steps: obtaining real-time rendering data of a first image frame before the rendering operation is executed on the first image frame; based on the real-time rendering data, identifying at least one of a picture zooming type corresponding to the first image frame, a carrier type used by a game role, a game scene and scene light complexity; determining an optimization strategy for the first image frame based on at least one of a picture zooming type corresponding to the first image frame, a carrier type used by a game role, a game scene and scene light complexity; and performing a rendering operation on the first image frame based on the optimization strategy. In this way, before the image is rendered, the picture scene which can influence the optimization strategy can be recognized in advance. The subsequent optimization algorithm can adopt the most appropriate optimization strategy to perform rendering operation on the first image frame based on the image condition recognized in advance, so that accurate rendering resource supply is realized, and the user experience is improved.

Description

Technical Field

[0001] This application belongs to the technical field of terminals, and particularly relates to a scene recognition method and an electronic device based on rendering data. Background Art

[0002] To meet the growing needs of users for game experiences, the game rendering scenes of game applications in electronic devices such as mobile phones are becoming increasingly rich and the rendering quality is getting higher. Correspondingly, the rendering power consumption of the electronic device also increases, thus requiring electronic devices such as mobile phones to support higher requirements for rendering resource supply.

[0003] However, current electronic devices such as mobile phones cannot achieve accurate rendering resource supply for game applications, resulting in poor user game experiences. Summary of the Invention

[0004] The embodiments of this application provide a scene recognition method and an electronic device based on rendering data, which can achieve accurate rendering resource supply and improve user experiences.

[0005] In a first aspect, this application provides a scene recognition method based on rendering data. The method is applied to an electronic device and includes: before performing a rendering operation on a first image frame, obtaining real-time rendering data of the first image frame; based on the real-time rendering data, identifying at least one of a screen scaling type corresponding to the first image frame, a vehicle type used by a game character, a game scene, and a scene light complexity; based on at least one of the screen scaling type corresponding to the first image frame, the vehicle type used by the game character, the game scene, and the scene light complexity, determining an optimization strategy for the first image frame; the optimization strategy includes at least one of the following: a system scheduling strategy, an interpolation strategy, an image super-resolution strategy, an image post-processing strategy, and a screen adjustment strategy; based on the optimization strategy, performing a rendering operation on the first image frame.

[0006] In this way, before performing a rendering operation on an image, the screen scene that can affect the optimization strategy can be identified in advance through the rendering data. For example, special situations such as a sudden enlargement of the screen, a game scene switch, a vehicle type used by a game character, and a high scene light complexity can be identified. In this way, the subsequent optimization algorithm can adopt the most appropriate optimization strategy based on the screen situation identified in advance to perform a rendering operation on the first image frame, achieve accurate rendering resource supply, and improve user experiences.

[0007] In one implementable manner, identifying the screen scaling type corresponding to the first image frame based on the real-time rendering data includes: determining, based on the real-time rendering data, a first distance between the game character corresponding to the first image frame and the camera; obtaining a second distance, where the second distance is the distance between the game character corresponding to the second image frame and the camera, and the second image frame is the previous image frame of the first image frame; determining that the screen scaling type corresponding to the first image frame is a screen scaling mutation when the difference between the first distance and the second distance is greater than a distance threshold; and determining that the screen scaling type corresponding to the first image frame is a stable screen scaling when the difference between the first distance and the second distance is less than or equal to the distance threshold.

[0008] In this way, before performing the rendering operation on the first image frame, it can be determined in advance whether the screen has a scaling mutation. Thus, corresponding optimization strategies can be matched in advance based on the screen scaling situation.

[0009] In one implementable manner, determining, based on the real-time rendering data, the first distance between the game character corresponding to the first image frame and the camera includes: obtaining, based on the real-time rendering data, the spatial coordinates of the game character corresponding to the first image frame and the spatial coordinates of the camera; determining, based on the spatial coordinates of the game character corresponding to the first image frame and a first offset, the spatial coordinates of the center of the sphere around which the camera rotates with respect to the game character; the first offset being the distance offset of the spatial coordinates of the game character relative to the spatial coordinates of the center of the sphere corresponding to the camera; and determining the first distance based on the spatial coordinates of the center of the sphere corresponding to the first image frame and the spatial coordinates of the camera.

[0010] In this way, determining the first distance based on the spatial coordinates of the game character and the camera can avoid the occurrence of incorrect recognition caused by the occlusion of different objects in the first image frame.

[0011] In one implementable manner, identifying the vehicle type used by the game character corresponding to the first image frame based on the real-time rendering data includes: when the first distance is the same as the second distance, determining a set of spatial coordinates of the center of the sphere around which the camera rotates with respect to the game character based on the spatial coordinates of the game character corresponding to the first image frame and a second set of offsets; wherein the second set of offsets includes the distance offsets of the spatial coordinates of the game character relative to the spatial coordinates of the center of the sphere corresponding to the camera when the game character uses different vehicles; determining a set of alternative distances based on the set of spatial coordinates of the center of the sphere and the spatial coordinates of the camera; determining a target distance from the set of alternative distances based on a first correspondence; the first correspondence being the correspondence between the vehicle type and the reference distance, the reference distance being the distance between the camera and the center of the sphere around which the camera rotates with respect to the game character when the game character uses a vehicle; the target distance belonging to the reference distances in the first correspondence; and determining the vehicle type used by the game character based on the first correspondence and the target distance.

[0012] In this way, the vehicle type used by the game character in each image frame can be matched in real time through the pre-established first correspondence.

[0013] In one implementable manner, identifying the game scene corresponding to the first image frame based on the real-time rendering data includes: obtaining the feature information of the static objects in the first image frame based on the real-time rendering data; determining the game scene corresponding to the first image frame based on the feature information of the static objects and a scene recognition database; the scene recognition database including a plurality of game scenes and the scene recognition features corresponding to each game scene in the plurality of game scenes.

[0014] In this way, the game scene corresponding to each image frame can be matched in real time through the pre-established scene recognition database.

[0015] In one implementable manner, the method further includes: obtaining historical rendering data, the historical rendering data including the rendering data corresponding to a plurality of game scenes; extracting the feature information of the static objects corresponding to the plurality of game scenes based on the historical rendering data; filtering the feature information of the static objects that can be destroyed to obtain alternative feature information; determining the scene recognition features corresponding to each game scene from the alternative feature information; and establishing a scene recognition database based on each game scene and the scene recognition features corresponding to each game scene.

[0016] In this way, by first filtering out the destructible static objects in each game scene, the feature information of the alternative static objects is obtained. Then, the scene recognition features corresponding to each game scene are further determined from the feature information of the alternative static objects. In this way, the situation of incorrect scene recognition caused by the destruction of the destructible static objects can be avoided in the subsequent process.

[0017] Among them, the scene recognition database can be stored in the system library of the electronic device. When the game scene switching recognition unit is used to recognize the game scene subsequently, the scene recognition database can be directly used to recognize and match the game scene without recalculation.

[0018] In an implementable manner, the recognizing the scene light complexity corresponding to the first image frame based on the real-time rendering data includes: determining the light source distance between each light source and the camera in the first image frame based on the real-time rendering data; determining the rendering weight corresponding to each light source based on the light source type, the light source distance of each light source in the first image frame, and a second corresponding relationship, where the second corresponding relationship is the corresponding relationship between the light source type, the light source distance, and the rendering weight; and determining the scene light complexity of the first image frame based on the rendering weight of each light source in the first image frame.

[0019] In this way, the second corresponding relationship can be pre-calculated through historical rendering data, and the scene light complexity corresponding to each image frame can be calculated in real time.

[0020] In an implementable manner, the method further includes: obtaining historical rendering data, where the historical rendering data includes historical light source information corresponding to each lighting scene, the position information of the historical camera, and historical rendering load information; the historical light source information includes the number of historical light sources, the type of historical light sources, and the position information of the historical light sources; determining the historical light source distance between each historical light source and the historical camera based on the position information of the historical light source and the position information of the historical camera; and determining the second corresponding relationship between the light source type, the light source distance, and the rendering weight based on the influence relationship between the historical light source distance and the historical rendering load information.

[0021] In an implementable manner, the obtaining the real-time rendering data of the first image frame includes: obtaining a real-time rendering instruction; and obtaining the rendering data corresponding to the rendering instruction when the rendering instruction includes a preset rendering instruction feature.

[0022] In this way, through the preset rendering instruction feature, only the rendering data required for scene recognition can be obtained, thereby reducing the workload of obtaining real-time rendering data.

[0023] In one implementable manner, an optimization strategy for the first image frame is determined based on at least one of the screen scaling type corresponding to the first image frame, the vehicle type used by the game character, the game scene, and the scene light complexity, including:

[0024] When the first image frame meets the first condition, it is determined that the optimization strategy for the first image frame includes binding the application thread corresponding to the first image frame to a core; the first condition includes at least one of the following: the game scene corresponding to the first image frame is the same as the game scene corresponding to the second image frame, the vehicle type used by the game character is the target vehicle type, the scene light complexity is greater than the light complexity threshold, where the second image frame is the previous image frame of the first image frame.

[0025] Among them, different vehicle types, game scenes, and scene light complexities can all affect the rendering complexity of performing rendering operations on the image. Among them, the target vehicle can be a vehicle type with a relatively large rendering overhead required.

[0026] In this way, the system scheduling algorithm can adjust the corresponding resource supply situation based on the vehicle type, game scene, and light complexity before performing the rendering process. For example, when it is recognized that the first image frame includes a target vehicle and / or the light complexity of the current lighting scene is relatively high, operations such as binding the game thread to a core and increasing the frequency can be performed in advance. In this way, when the rendering operation is actually performed, based on the operations such as binding the core and increasing the frequency set in advance, the running speed of high-load game scenes can be optimized.

[0027] When the first image frame meets the second condition, it is determined that the optimization strategy for the first image frame includes rendering the first image frame using an interpolation algorithm and / or an image super-resolution algorithm; the second condition includes at least one of the following: the screen scaling type is screen stable scaling, the game scene corresponding to the first image frame is the same as the game scene corresponding to the second image frame, and the second image frame is the previous image frame of the first image frame.

[0028] In this way, the interpolation algorithm can learn in advance the difference between the current image frame and the previous image frame before performing the rendering process. Furthermore, the interpolation algorithm can decide whether to perform interpolation processing before performing the rendering process. In this way, when the rendering operation is actually performed, interpolation processing can be performed or not performed based on the interpolation processing decision. For example, when the screen scaling image type issued by the screen scaling recognition unit is screen scaling mutation, the interpolation algorithm can decide not to perform interpolation processing on the current image frame to avoid obtaining a prediction frame with poor image quality due to a large difference between the two frames of images. In this way, when the rendering operation is actually performed, the rendering prediction frame will not be drawn, thereby saving the computing power overhead of the interpolation algorithm.

[0029] When the game scene corresponding to the first image frame is a low-light game scene, determining the optimization strategy for the first image frame includes increasing the screen brightness of the electronic device; when the game scene corresponding to the first image frame is such that the requirement for followability is greater than the followability threshold, determining the optimization strategy for the first image frame includes reducing the screen refresh rate; when the game scene corresponding to the first image frame is such that the requirement for followability is less than or equal to the followability threshold, determining the optimization strategy for the first image frame includes increasing the screen refresh rate.

[0030] In this way, the post-processing algorithm or the screen adjustment algorithm can dynamically adjust the screen brightness based on the recognized game scene. For example, if the game scene switching recognition unit recognizes that the game scene corresponding to the current image frame is a darker underground scene, the screen brightness can be increased through the post-processing algorithm or the screen adjustment algorithm. The screen adjustment algorithm can also dynamically adjust the screen refresh rate based on the game scene to provide a better gaming experience for the user.

[0031] In one implementable manner, the method further includes: obtaining configuration data of the game application corresponding to the first image frame; determining one or more of the following parameters based on the configuration data, where the parameters include a first offset, a second set of offsets, a first correspondence, a scene recognition database, and a second correspondence.

[0032] In this way, the present application can support the recognition of the screen scenes of multiple games and is more universal.

[0033] In one implementable manner, the configuration data includes the game name; determining one or more of the following parameters based on the configuration data includes: determining one or more of the parameters based on the game name.

[0034] In a second aspect, the present application provides an electronic device, including a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method according to any one of the first aspects.

[0035] In a third aspect, the present application provides a chip system, characterized in that the chip system includes a processor; the processor is coupled to a memory, and the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the method according to any one of the first aspects is executed.

[0036] Fourthly, the present application provides a computer-readable storage medium, characterized in that a computer program or instruction is stored in the computer-readable storage medium, and when the computer program or instruction runs on a computer, the computer is enabled to execute the method according to any one of the first aspect. Description of the Drawings

[0037] Figure 1 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application;

[0038] Figure 2 It is a block diagram of the software structure of an electronic device provided by an embodiment of the present application;

[0039] Figure 3 It is a flowchart of a method for scene recognition based on rendering data provided by an embodiment of the present application;

[0040] Figure 4 It is a flowchart of a method for intercepting rendering data provided by an embodiment of the present application;

[0041] Figure 5 It is a schematic diagram of the structure of a scene recognition module provided by an embodiment of the present application;

[0042] Figure 6 It is a schematic diagram of the movement of a camera around the main character in a game world provided by an embodiment of the present application;

[0043] Figure 7 It is a scene diagram of a sudden change in screen scaling provided by an embodiment of the present application;

[0044] Figure 8 It is a working flowchart of a method for scene recognition of screen scaling type provided by an embodiment of the present application;

[0045] Figure 9 It is a flowchart of a method for vehicle recognition provided by an embodiment of the present application;

[0046] Figure 10 It is a working flowchart of a method for vehicle recognition provided by an embodiment of the present application;

[0047] Figure 11 It is a working flowchart of a method for game scene recognition provided by an embodiment of the present application;

[0048] Figure 12 It is a working flowchart of a method for scene light complexity recognition provided by an embodiment of the present application;

[0049] Figure 13 It is a flowchart of a method for scene recognition based on rendering data provided by an embodiment of the present application;

[0050] Figure 14Flowchart of another scene recognition method based on rendering data provided by an embodiment of this application;

[0051] Figure 15 Block diagram of a chip provided by an embodiment of this application. Detailed implementation manners

[0052] To meet the growing needs of users for game experience, the game rendering scenes of game applications in electronic devices such as mobile phones are becoming increasingly rich and the rendering quality is getting higher and higher. The memory occupancy and rendering power consumption in the electronic device also increase accordingly. However, due to the power consumption of the electronic device or the capacity limitations of the CPU and GPU, high refresh rates and high frame rates often cause the electronic device to heat up or freeze, thus affecting the user experience.

[0053] Taking mobile games as an example, with the development and popularization of large mobile games, the rendering pipeline of modern mobile games has become increasingly complex and the resource load has become larger. However, the computing power growth of modern mobile phones does not meet the needs of modern large mobile games, and mobile phones are limited by the limited battery capacity and heat dissipation capacity. Therefore, to improve the user's game experience, many optimization algorithms have emerged, such as system resource scheduling algorithms, frame interpolation algorithms, image super-resolution algorithms, etc.

[0054] However, currently these optimization algorithms are all optimized based on the rendered image frames. Taking the frame interpolation algorithm as an example, the frame interpolation algorithm generates a prediction frame based on two adjacent rendered image frames. In this way, the refresh rate can be increased by inserting a prediction frame between two adjacent image frames. However, when the difference between two adjacent rendered image frames is large, problems such as image distortion and deformation will occur in the generated prediction frame.

[0055] To solve the above technical problems, an embodiment of this application provides a scene recognition method based on rendering data. Before performing a rendering operation on an image, the rendering data is used to identify the screen scene corresponding to the current game application in advance. In this way, subsequent optimization algorithms can execute the best optimization strategy based on the pre-identified scene changes, improving the user experience.

[0056] Exemplarily, if it is recognized that the difference between the image frame to be rendered and the previous image frame is large before performing a rendering operation on the image, the frame interpolation algorithm can decide not to perform frame interpolation before performing a rendering operation on the image. In this way, on the one hand, it can avoid generating prediction frames with problems such as image distortion and deformation. On the other hand, it can save the computing power overhead of the frame interpolation algorithm.

[0057] The scene recognition method based on rendering data provided by the embodiments of this application can be applied to electronic devices with a display function. The electronic device can be a mobile phone, smart TV, wearable device, tablet computer (Pad), computer with wireless transceiver function, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, and so on.

[0058] In the embodiments of this application, the terminal device can also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc.

[0059] One or more game applications can be installed in the electronic device. When the electronic device runs the application program of the game application, the electronic device processes and displays the corresponding game interface based on the scene recognition method based on rendering data of the embodiments of this application.

[0060] Exemplarily, when the electronic device receives a click operation from the user for the game application, the electronic device starts to run the game application and processes and displays the corresponding game interface based on the scene recognition method based on rendering data of the embodiments of this application.

[0061] Taking the electronic device as a mobile phone as an example, the structure of the electronic device in the embodiments of this application will be described below.

[0062] Figure 1 It is a schematic diagram of the hardware structure of an electronic device provided by the embodiments of this application. As Figure 1As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0063] It can be understood that the structure schematically shown in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0064] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0065] The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0066] A memory can also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0067] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface 130, etc.

[0068] The charging management module 140 is configured to receive a charging input from a charger. The charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input from the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0069] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0070] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0071] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0072] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through the antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0073] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0074] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive the signals to be transmitted from the processor 110, perform frequency modulation on them, amplify them, and convert them into electromagnetic waves through the antenna 2 and radiate them out.

[0075] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0076] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0077] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0078] The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc.

[0079] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera photosensitive element. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0080] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0081] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0082] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0083] The NPU is a neural-network (NN) computing processor. By learning from the biological neural network structure, such as learning from the transmission mode between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0084] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0085] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor.

[0086] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0087] The button 190 includes a power-on button, volume buttons, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs and generate key signal inputs related to the user settings and function controls of the electronic device 100.

[0088] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. For touch operations on different regions of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminders, receiving messages, alarms, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0089] The indicator 192 can be an indicator light and can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0090] The SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to achieve functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0091] The software system of the electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservices architecture, or cloud architecture. In the embodiments of the present invention, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device 100.

[0092] Figure 2 It is a software structure block diagram of the electronic device 100 in the embodiments of the present application.

[0093] The layered architecture divides software into several layers, and each layer has clear roles and divisions of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer (HAL), and the kernel layer.

[0094] The application layer may include a series of application packages.

[0095] As Figure 2 shown, the application packages may include applications such as camera, gallery, calendar, call, map, game, WLAN, Bluetooth, music, video, short message, etc.

[0096] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0097] As Figure 2 shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0098] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0099] The content provider is used to store and obtain data, and make this data accessible to applications. The data may include video, image, audio, dialed and answered calls, browsing history and bookmarks, phone book, etc.

[0100] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon may include a view for displaying text and a view for displaying pictures.

[0101] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call status (including answering, hanging up, etc.).

[0102] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, and so on.

[0103] The notification manager enables an application to display notification information in the status bar. It can be used to convey messages of the notification type, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that a download is complete, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background-running application, or a notification that appears on the screen in the form of a dialog window. For example, it can prompt text information in the status bar, emit a prompt sound, vibrate the electronic device, blink the indicator light, etc.

[0104] The Android Runtime includes the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0105] The core libraries consist of two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of Android.

[0106] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as the management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0107] The system libraries can include multiple functional modules. For example: surface manager, Media Libraries, 3D graphics processing library (such as: OpenGL ES), 2D graphics engine (such as: SGL), etc.

[0108] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0109] The media library supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0110] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0111] The 2D graphics engine is a graphics engine for 2D drawing.

[0112] In the embodiments of this application, the system libraries can also include optimization algorithms such as a scene recognition module, a scene recognition service, a system scheduling algorithm, an interpolation algorithm, an image super-resolution algorithm, a post-processing algorithm, and a screen regulation algorithm.

[0113] Among them, the scene recognition module can be used to recognize in advance the game scene, sudden change in screen scaling, vehicle type, light complexity, etc. corresponding to the image frame to be rendered based on the rendering data.

[0114] Optimization algorithms such as the frame interpolation algorithm, image super-resolution algorithm, post-processing algorithm, and screen adjustment algorithm can subscribe to the corresponding scene recognition service. Then, the scene recognition module can distribute the corresponding recognition results to each optimization algorithm based on the subscription situation of each optimization algorithm. In this way, each optimization algorithm can make algorithm decisions such as resource adjustment in advance based on the recognition results.

[0115] HAL is an abstract interface for the device kernel driver, which realizes providing an application programming interface for accessing the underlying device to a higher-level java API framework. The hardware abstraction layer can include multiple library modules, for example, a display module, an audio module, a Bluetooth module, a Wi-Fi module, etc. Each module can implement an interface for a specific type of hardware component. When the framework API requests access to the device hardware, the Android system will load the library module for this hardware component.

[0116] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0117] Next, the scene recognition method based on rendering data provided by the embodiments of the present application will be described.

[0118] As Figure 3 shown, the scene recognition method based on rendering data provided by the embodiments of the present application mainly includes two parts. The first part is to calculate the basic parameters required by the scene recognition module based on historical rendering data. The second part is to use the scene recognition module to recognize the screen scene based on real-time rendering data and the calculated basic parameters when the game application is running.

[0119] In some embodiments, calculating the basic parameters required by the scene recognition module based on historical rendering data can be implemented in the following manner: various game scenes, interaction modes, game special effects, etc. in the game can be run in advance to obtain various rendering data required for calculating the basic parameters.

[0120] In some embodiments, when the game application is run historically, the electronic device can save the corresponding rendering data to the system library of the electronic device (for example, the local system library), so as to obtain historical rendering data.

[0121] Exemplarily, as Figure 4As shown, in response to receiving user clicks, slides, and other operations on the game application interface, the game application at the application layer calls the rendering engine at the framework layer, and sends rendering instructions to the system library through the rendering engine. Based on the received rendering instructions, the system library prepares rendering resources corresponding to the rendering instructions. After that, the system library sends the rendering resources to the GPU, and the GPU renders the game screen based on the rendering resources. After the GPU obtains the rendered game screen, it can output it to the display screen for displaying the game screen.

[0122] The embodiment of the present application can intercept resources during the rendering resource preparation stage to obtain rendering data for the GPU to render the game screen, wherein the obtained rendering data can be stored in the system library of the electronic device.

[0123] It should be noted that in the embodiments of the present application, both historical rendering data and real-time rendering data can be obtained using the above-mentioned interception method.

[0124] The historical rendering data may include rendering data corresponding to multiple frames of game screens. The rendering data corresponding to each frame of the game screen may include a variety of data. For example, the rendering data may include texture information, vertex data, camera space coordinates, static object space coordinates, dynamic object space coordinates, light source space coordinates, light source number, light source type, etc.

[0125] The embodiment of the present application can statistically analyze and calculate the rendering data required for the basic parameters of the scene recognition module based on the historical rendering data, that is, the target rendering data. Then, the target rendering data can be extracted from the numerous historical rendering data.

[0126] In some embodiments, the characteristics of the rendering instructions corresponding to each target rendering data may also be statistically analyzed. In this way, in the subsequent real-time running process of the game, the corresponding target rendering data may be intercepted based on the characteristics of the rendering instructions.

[0127] Exemplarily, a correspondence between the rendering instruction feature and the target rendering data may be established. For example, the correspondence includes a feature tag corresponding to the rendering instruction feature and the target rendering data. In this way, when it is recognized that the rendering instruction includes a feature corresponding to the feature tag, the target rendering data corresponding to the rendering instruction is intercepted.

[0128] In this way, after the target rendering data is obtained, the basic parameters corresponding to the scene recognition module can be calculated based on the target rendering data.

[0129] like Figure 5 As shown, the scene recognition module in the embodiment of the present application may include four recognition units: a screen zoom recognition unit, a game scene switching recognition unit, a vehicle recognition unit and a scene light complexity recognition unit.

[0130] Among them, the screen zoom recognition unit can be used to recognize the degree of screen magnification or reduction based on the rendering data. If the degree of screen magnification or reduction is too large, it can be considered that the screen zoom mutates.

[0131] The game scene switching recognition unit can be used to recognize the game scene corresponding to each frame of the image based on the rendering data, and then it can be determined whether the game scene of the current image frame changes relative to the previous image frame. For example, the game scene can include a lobby game scene, a land game scene, an underwater game scene, etc.

[0132] The vehicle recognition unit can be used to recognize the tool that can enable the character in the game to move by driving or riding based on the rendering data. For example, the vehicle can include a vehicle, a ship, an airplane, a tank, etc.

[0133] The scene light complexity recognition unit can be used to recognize the light complexity corresponding to each frame of the image based on the rendering data.

[0134] Among them, the target rendering data and the corresponding basic parameters to be calculated required by the four recognition units, namely the screen zoom recognition unit, the vehicle recognition unit, the game scene switching recognition unit, and the scene light complexity recognition unit, in the scene recognition module are different.

[0135] The calculation methods of the basic parameters of the above four recognition units and the methods of using the four recognition units for scene recognition will be described below.

[0136] The screen zoom recognition unit provided in the embodiment of the present application will be described first below.

[0137] As Figure 5 shown, the target rendering data required to calculate the basic parameters corresponding to the screen zoom recognition unit includes: the spatial coordinates of the main character and the spatial coordinates of the camera.

[0138] Among them, the spatial coordinates of the main character refer to the position information of the main character in the game world. The main character can be understood as the character controlled by the game player.

[0139] The camera (which can also be called a follow camera) in the game environment is a virtual camera, and the position of the camera can reflect the perspective of the game player in the game screen. This perspective will automatically move or rotate following and around the main character controlled by the game player.

[0140] Exemplarily, as Figure 6As shown, in game design, the camera 10 always moves spherically around the main character 20. Among them, the spatial coordinates of the center of the sphere C1 around which the camera 10 moves spherically around the main character 20 are usually not the same as the spatial coordinates P1 corresponding to the main character 20. The distance R1 between the center of the sphere C1 and the camera 10 represents the radius of the rotating sphere. Among them, the larger the value of R1, the wider the range of the game world that the camera 10 can display and the farther the perspective, but the relatively poor display of the details of the game characters. On the contrary, the smaller the value of R1, the smaller the range of the game world that the camera 10 can display, the closer the perspective, and the more prominent the details of the game characters.

[0141] In some embodiments, the size of the value of R1 can change in response to the user's operation of zooming in or out on the game screen. For example, after the user performs a zoom-in operation on the game screen, the value of R1 will become larger. After the user performs a zoom-out operation on the game screen, the value of R1 will become smaller.

[0142] In some embodiments, the user cannot control the zooming in or out of the game screen. That is to say, the distance between the camera 10 and the main character 20 in the game remains constant unless in the camera collision scenario, the distance between the camera and the main character will be automatically reduced.

[0143] Exemplarily, in combination with Figure 6 and Figure 7 as shown, when the camera is at M1, the frame game screen is the screen shown in Figure 7 (a). As the main character 20 rotates, the camera 10 rotates around the main character 20. When the perspective of the camera is blocked by the wall in Figure 7 (a), the main character 10 will not be visible. In this case, the distance between the camera 10 and the main character 20 can be reduced, and the perspective of the camera 10 can be enlarged to ensure that the main character 20 is in the screen. For example, the camera 10 can be advanced from M1 to M1', and the switched screen is the screen shown in Figure 7 (b). Among them, the scenario of switching from the screen shown in Figure 7 (a) to the screen shown in Figure 7 (b) is a screen zoom mutation scenario. This screen mutation scenario can also be called a camera collision scenario.

[0144] In the embodiments of the present application, the screen zoom mutation scenario refers to a scenario where the distance between the camera 10 and the main character 20 suddenly changes greatly, so that the scaling difference between the current frame game screen and the previous frame game screen is relatively large.

[0145] Figure 8 This is a flowchart of the working process of a method for identifying a screen zoom scenario provided by the embodiments of the present application. As Figure 8 shown, it may include the following steps:

[0146] S201. Obtain the spatial coordinates of the main character and the spatial coordinates of four non - coplanar cameras when the position of the main character remains unchanged at the farthest viewing angle from historical rendering data.

[0147] Exemplarily, please continue to refer to Figure 6 , obtain the spatial coordinates of the main character at four different times t1, t2, t3, and t4 and the spatial coordinates of the camera 10 when the position of the main character 20 remains unchanged at the farthest viewing angle. Among them, at times t1, t2, t3, and t4, the position of the main character 20 is the same. For example, the spatial coordinates of the main character 20 are all P1(x0, y0, z0). At times t1, t2, t3, and t4, the camera 10 rotates around the main character 20 on the same sphere, that is, the position of the camera 10 is different. For example, at time t1, the spatial coordinates of the camera 10 are M1(x1, y1, z1); at time t2, the spatial coordinates of the camera 10 are M2(x2, y2, z2); at time t3, the spatial coordinates of the camera 10 are M3(x3, y3, z3); at time t4, the spatial coordinates of the camera 10 are M4(x4, y4, z4). Among them, the positions of the camera 10 at times t1, t2, t3, and t4 are non - coplanar.

[0148] S202. Calculate the spatial coordinates C1 of the center of the maximum sphere and the radius R1 based on the spatial coordinates of the four non - coplanar cameras.

[0149] Exemplarily, based on the spatial coordinates M1, M2, M3, and M4 of the camera, the sphere equation can be solved using the following formula (1) to obtain the spatial coordinates C1(a, b, c) of the center of the sphere and the radius R1.

[0150] (x - a) 2 +(y - b) 2 +(z - c) 2 =R1 2 Formula (1)

[0151] Where x, y, z represent the spatial coordinates of the camera, a, b, c represent the spatial coordinates of the center of the sphere, and R1 represents the radius of the maximum sphere.

[0152] S203. Calculate the offset offset1 between the center of the sphere and the main character based on the spatial coordinates C1 of the center of the sphere and the spatial coordinates P1 of the main character.

[0153] It should be understood that after the position of the main character changes in the game world, the spatial coordinates P1 of the main character will change, but the offset offset1 between the center of the sphere and the main character will not change.

[0154] In this way, the maximum sphere radius R1 and the offset offset1 obtained by the above method can be used as the basic parameters of the screen mutation scene recognition unit. When subsequently using the screen scaling recognition unit to identify whether the game screen mutates, these basic parameters can be directly used without recalculation. For example, the above basic parameters can be saved in the system library of the electronic device.

[0155] It should be noted that the above embodiments are only exemplarily described by taking the calculation of a set of basic parameters as an example, and do not represent a limitation on the number of basic parameters. For example, in the embodiments of the present application, for multiple games, the corresponding maximum sphere radius R1 and offset offset1 can be calculated respectively. Then, the calculated multiple sets of sphere radius R1 and offset offset1 are used as basic data and saved in the system library of the electronic device for use when subsequently using the screen scaling recognition unit to identify whether the game screen scales and mutates. In this way, the embodiments of the present application can support the recognition of screen scaling mutation scenes of multiple games and are more universal.

[0156] For the method of performing screen scaling recognition using the above basic parameters of sphere radius R1 and offset offset1, please refer to the description of steps S204 to S210.

[0157] S204, intercept the rendering data of the current image frame to obtain the spatial coordinates P2 of the main character and the spatial coordinates M5 of the camera.

[0158] In the embodiments of the present application, the current image frame refers to the image frame to be rendered. Exemplarily, the previous image frame refers to the image frame displayed on the current screen, and the current image frame refers to the next image frame to be displayed.

[0159] S205, based on the spatial coordinates P2 of the main character and the offset offset1 in the basic parameters, determine the spatial coordinates C2 of the center of the sphere around which the camera rotates relative to the main character.

[0160] S206, based on the spatial coordinates C2 of the center of the sphere and the spatial coordinates M5 of the camera, determine the sphere radius R_D corresponding to the current image frame.

[0161] S207, based on the sphere radius R_D corresponding to the current image frame and the sphere radius corresponding to the previous image frame, determine the change amount A of the camera relative to the center of the sphere.

[0162] Exemplarily, such as Figure 6As shown, the spatial coordinates of the main character corresponding to the current image frame are P2, and the spatial coordinates of the camera are M5. Although the position of the main character, the position of the camera, and the position of the center of the sphere around which the camera rotates relative to the main character will change with the change of the position of the main character, the offset between the center of the sphere C2 around which the camera rotates relative to the main character and the spatial coordinates P2 of the main character remains constant.

[0163] Therefore, based on the spatial coordinates P2 of the main character and the offset offset1 in the basic parameters, the spatial coordinates C2 of the center of the sphere corresponding to the current image frame can be determined. Further, based on the spatial coordinates C2 of the center of the sphere and the spatial coordinates M5 of the camera, the sphere radius R_D corresponding to the current image frame can be determined. The sphere radius R_D refers to the distance between the camera and the main character in the current image frame.

[0164] In some embodiments, for games where the user cannot control the zooming in or out of the game screen, the sphere radius corresponding to the previous image frame in step S207 is always the maximum sphere radius R1.

[0165] In some embodiments, for games where the value of R1 can change in response to the user's zooming in or out operation on the game screen, generally, when the game is first entered, the distance between the camera and the main character is the farthest. Therefore, the sphere radius corresponding to the first image frame after entering the game is the maximum sphere radius R1. Thus, if the current image frame is the second image frame after the first image frame, the change amount A of the current camera relative to the center of the sphere is obtained based on the difference between the sphere radius R_D corresponding to the current image frame and the maximum sphere radius R1.

[0166] If the current image frame is the third image frame, the change amount A of the current camera relative to the center of the sphere is obtained based on the difference between the sphere radius R_D corresponding to the current image frame and the sphere radius R_D corresponding to the second image frame.

[0167] Similarly, the above method can be used to determine the change amount A corresponding to subsequent image frames, which will not be elaborated here.

[0168] S208, determine whether the change amount A is greater than the distance threshold.

[0169] S209, in the case where the change amount A is less than or equal to the distance threshold, identify the current scene as a stable screen zooming scene.

[0170] Exemplarily, for games where the value of R1 can change in response to the user's zooming in or out operation on the game screen, its zooming in or out operation belongs to a smooth zooming change scene, and generally there will be no sudden screen magnification. Therefore, for such games, generally the change amount A is less than or equal to the distance threshold, that is, the current scene is a stable screen zooming scene.

[0171] S210. When the change amount A is greater than the distance threshold, identify the current scene as a scene of sudden change in screen zoom.

[0172] Exemplarily, for a game where the user cannot control the zooming in or out of the game screen, when a camera collision scene as shown in Figure 7 occurs, the value of R1 will suddenly decrease, resulting in the change amount A being greater than the distance threshold. From the user's perspective, the game screen is instantaneously magnified. This situation can be identified as a scene of sudden change in screen zoom.

[0173] In this way, the screen zoom recognition unit provided in the embodiments of the present application can, based on real-time rendering data, obtain that the scene of sudden change is a scene of stable screen zoom or a scene of sudden change in screen zoom.

[0174] It should be noted that the change amount A in the above embodiments is an instantaneous value, that is, the change amount corresponding to one image frame. The embodiments of the present application can also calculate whether the sphere radius R_D corresponding to the current image frame is a stable value based on the method provided in the above embodiments.

[0175] Exemplarily, if the sphere radii R_D corresponding to a plurality of consecutive image frames before the current image frame are the same, it can be determined that the sphere radius R_D corresponding to the current image frame is a stable value.

[0176] Next, the vehicle recognition unit provided in the embodiments of the present application will be described.

[0177] First of all, it should be noted that before and after the main character drives or rides in a vehicle, the corresponding viewing angle range will change. However, when the main character drives or rides in a vehicle, the viewing angle range will be stable within the viewing angle range corresponding to the main character driving or riding in the vehicle. Therefore, when it is detected that the sphere radius R_D changes and is stable at the changed sphere radius R_D, the vehicle driven or ridden by the main character can be further recognized.

[0178] Exemplarily, as shown in Figure 9 , when it is detected that the sphere radius R_D of the i-th image frame changes and the sphere radii R_D of N image frames after the i-th image frame are the same as the sphere radius R_D of the i-th image frame, it can be determined that the sphere radius R_D is a stable value. In this way, when the sphere radius R_D is a stable value, the vehicle recognition unit can perform vehicle type recognition. For example, the vehicle types can include vehicle 1, vehicle 2, vehicle 3, etc. Finally, the vehicle recognition unit can output the recognized vehicle type. Wherein, i is a positive integer and N is a positive integer greater than 0.

[0179] Among them, the method for calculating the sphere radius R_D of each image frame can refer to the description in steps S204 to S210, which will not be elaborated here.

[0180] Secondly, it should be noted that when the main character drives or rides in a vehicle, the camera still rotates around the main character. However, the perspectives corresponding to the main character driving or riding in different vehicles are different. Therefore, the center of the sphere and the radius of the sphere corresponding to the main character driving or riding in different vehicles are different.

[0181] Based on this, the target rendering data required for calculating the basic parameters of the vehicle recognition unit in the embodiments of the present application may include: the spatial coordinates of the main character, the spatial coordinates of the camera, and the spatial coordinates of the vehicle when the main character drives or rides in different vehicles.

[0182] Figure 10 It is a flowchart of a vehicle recognition method provided by an embodiment of the present application. As Figure 10 shown, it may include the following steps:

[0183] S301, obtain the spatial coordinates of the main character and the spatial coordinates of four non-coplanar cameras when the main character drives or rides in a vehicle from the historical rendering data.

[0184] S302, calculate the spatial coordinates C3 of the center of the sphere and the radius R2 of the sphere corresponding to the current vehicle based on the spatial coordinates of the four non-coplanar cameras.

[0185] S303, calculate the offset offset2 between the spatial coordinates C3 of the center of the sphere and the main character P3 based on the spatial coordinates C3 of the center of the sphere corresponding to the current vehicle and the spatial coordinates of the main character P3.

[0186] Traverse all vehicles that need to be recognized in the game, and repeat the above steps S301 to S303, and the spatial coordinates C3 of the center of the sphere corresponding to each vehicle and the offset offset2 between the spatial coordinates C3 and the main character P3 can be obtained.

[0187] Among them, the specific implementation processes of steps S301 to S303 can refer to the descriptions of steps S201 to S203, which will not be elaborated here.

[0188] In this way, the sphere radius R2 and the offset offset2 corresponding to each vehicle obtained by the above method can be used as the basic parameters of the vehicle detection and recognition unit. When using the vehicle detection and recognition unit to recognize vehicles in the game subsequently, these basic parameters can be directly used without recalculation. For example, the above basic parameters can be saved in the system library of the electronic device.

[0189] It should be noted that the above embodiments are only exemplary described by taking the calculation of a set of basic parameters as an example, and do not represent a limitation on the number of basic parameters. For example, the embodiments of the present application can calculate the sphere radius R2 and the offset offset2 corresponding to each vehicle in multiple games respectively. Then, the multiple sets of sphere radius R2 and offset offset2 corresponding to the multiple games obtained by calculation are used as basic data and stored in the system library of the electronic device for use when the vehicle detection and recognition unit recognizes the vehicle in the game later. In this way, the embodiments of the present application can support the recognition of vehicles in multiple games and have better universality.

[0190] It should also be noted that the above embodiments are only exemplary described by taking the offset offset2 between the spatial coordinates C3 of the center of the sphere corresponding to each vehicle and the main character P3 as the basic data, and do not represent a limitation on the basic data corresponding to the vehicle detection and recognition unit.

[0191] During the rotation of the camera along with the main character, the offset offset2 between the spatial coordinates C3 of the center of the sphere corresponding to the vehicle and the main character P3 will not change, and the offset offset3 between the spatial coordinates C3 of the center of the sphere corresponding to the vehicle and the vehicle will not change either. Therefore, the embodiments of the present application can also use the offset offset3 between the spatial coordinates C3 of the center of the sphere corresponding to each vehicle and the vehicle as the basic data. In this case, when obtaining the spatial coordinates of the vehicle and the spatial coordinates of four non-coplanar cameras when the main character drives or rides in the vehicle in step S301. Then, in step S302, based on the spatial coordinates C3 of the center of the sphere corresponding to the current vehicle and the spatial coordinates of the current vehicle, calculate the offset offset3 between the spatial coordinates C3 of the center of the sphere and the spatial coordinates of the current vehicle.

[0192] Then, the above basic parameters (the sphere radius R2 and the offset offset2 corresponding to each vehicle) can be used for vehicle recognition. For the specific vehicle recognition method, please refer to the description in steps S304 to S306.

[0193] S304, intercept the rendering data of the current image frame to obtain the spatial coordinates P4 of the main character and the spatial coordinates M6 of the camera when the main character drives or rides in the vehicle.

[0194] S305, based on the spatial coordinates P4 of the main character and the offsets offset2 in the basic parameters, determine the spatial coordinates C4 of the centers of the spheres around which the multiple cameras rotate around the main character.

[0195] S306, based on the spatial coordinates C4 of the centers of the spheres and the spatial coordinates M6 of the cameras, determine multiple sphere radii R_D.

[0196] Among them, steps S304 to S306 can refer to the descriptions of steps S301 to S303, which will not be elaborated here.

[0197] S307. Based on the corresponding relationship between each vehicle and the sphere radius R2 and each sphere radius R_D, determine the vehicle type used by the main character in the current image frame.

[0198] Since the sphere radius R2 corresponding to each vehicle has been obtained in advance based on historical rendering data, the sphere radius R_D corresponding to the current image frame can be matched with the sphere radius R2 corresponding to each vehicle to determine the vehicle corresponding to the current image frame.

[0199] Exemplarily, as shown in Table 1, the vehicles in the game include cars, airplanes, ships, and tanks. Among them, the sphere radius corresponding to the main character driving a car is R2-1, the sphere radius corresponding to the main character driving an airplane is R2-2, the sphere radius corresponding to the main character driving a ship is R2-3, and the sphere radius corresponding to the main character driving a tank is R2-4. In this way, if the sphere radius R_D calculated in step S306 includes R2-1, it can be determined that the vehicle driven by the main character is a car. If the sphere radius R_D calculated in step S306 includes R2-2, it can be determined that the vehicle driven by the main character is an airplane. If the sphere radius R_D calculated in step S306 includes R2-3, it can be determined that the vehicle driven by the main character is a ship. If the sphere radius R_D calculated in step S306 includes R2-4, it can be determined that the vehicle driven by the main character is a tank.

[0200] Table 1 Corresponding relationship between vehicle and sphere radius

[0201] Vehicle type Sphere radius Automobile R2-1 Airplane R2-2 Ship R2-3 Tank R2-4

[0202] The game scene switching recognition unit provided by the embodiments of the present application will be described below.

[0203] A complete game usually includes multiple game scenes, and different game scenes can correspond to different game environments. Among them, the game environment usually includes static objects such as buildings, trees, vegetation, and terrain. The embodiments of the present application can identify the game scene by analyzing the characteristics of the static objects in different game scenes.

[0204] In this way, the target rendering data required to calculate the basic parameters corresponding to the game scene switching recognition unit includes: the spatial coordinates of the static objects.

[0205] Figure 11 is the workflow diagram of a game scene switching recognition method provided by the embodiments of the present application. As Figure 11 shown, it can include the following steps:

[0206] S401. Obtain the spatial coordinates of static objects in all game scenes from historical rendering data.

[0207] S402. Filter the spatial coordinates of destructible static objects to obtain the spatial coordinates of alternative static objects.

[0208] S403. Determine the scene recognition features corresponding to each game scene from the spatial coordinates of alternative static objects.

[0209] In the embodiments of the present application, the destructible static objects in each game scene can be filtered first to obtain the spatial coordinates of alternative static objects. Then, further determine the scene recognition features corresponding to each game scene from the spatial coordinates of the alternative static objects. In this way, it is possible to avoid the situation of incorrect scene recognition caused by the destruction of destructible static objects subsequently.

[0210] In some embodiments, the most representative static objects in each game scene can be determined from the spatial coordinates of alternative static objects. Then, the spatial coordinates of each most representative static object are respectively determined as the scene recognition features of each game scene.

[0211] Among them, the most representative static object refers to a static object unique to each game scene that can be distinguished from other game scenes. For example, the most representative static object can be one or several of buildings, trees, vegetation, and terrain.

[0212] In this way, a scene recognition database can be obtained based on each game scene and the corresponding scene recognition features. Among them, the scene recognition database can be stored in the system library of the electronic device. When the game scene switching recognition unit is used to recognize the game scene subsequently, the scene recognition database can be directly used to recognize and match the game scene without recalculation.

[0213] It should be noted that the above embodiments are only used for exemplary illustration by taking the construction of a scene recognition database for a game as an example. The above method can also be used to construct scene recognition databases corresponding to multiple games and store them in the system library of the electronic device. In this way, the embodiments of the present application can support the scene recognition of multiple games and have greater universality.

[0214] Furthermore, real-time game scene recognition can be performed based on the established scene recognition database. For specific details, please refer to the descriptions of steps S404 to S405.

[0215] S404. Intercept the rendering data of the current image frame to obtain the spatial coordinates of static objects.

[0216] S405. Determine the game scene corresponding to the current image frame based on the spatial coordinates of the matching static objects corresponding to the current image frame and the scene recognition database.

[0217] Exemplarily, the spatial coordinates of the static objects corresponding to the current image frame can be compared with the scene recognition features corresponding to each game scene in the scene recognition database. If the spatial coordinates of the static objects corresponding to the current image frame match the scene recognition features corresponding to a certain game scene, it can be determined that the game scene of the current image frame is the game scene that matches it.

[0218] Furthermore, it is also possible to determine whether the game scene has been switched based on the game scene corresponding to the current image frame and the game scene corresponding to the previous image frame.

[0219] The scene light complexity recognition unit provided by the embodiments of the present application will be described below.

[0220] In game design, a light source is a virtual light source set to simulate and render lighting effects. Designers can place different numbers and types of light sources in the scene according to requirements to create an ideal lighting and atmosphere effect.

[0221] The target rendering data required for the embodiments of the present application to calculate the basic parameters of the scene light complexity recognition unit may include: the spatial coordinates of the light sources corresponding to each lighting scene, the number of light sources, the type of light sources, the spatial coordinates of the camera, and the rendering load information.

[0222] Figure 12 It is a flowchart of the working process of a scene light complexity recognition method provided by the embodiments of the present application. As Figure 12 shown, it may include the following steps:

[0223] S501, obtain the light source information, rendering load information, and the spatial coordinates of the camera corresponding to the lighting scene from the historical rendering data.

[0224] The light source information may include the number of light sources, the type of light sources, and the spatial coordinates of the light sources. Among them, the type of light sources may include point light sources, directional lights, spotlights, etc.

[0225] The rendering load information refers to the rendering load index corresponding to rendering a picture under the lighting scene. For example, the rendering load index may include: the idle degree, load degree, frequency point of the GPU, etc., and may also include the frequency point of the CPU and the memory frequency point, etc.

[0226] In the embodiments of the present application, the rendering load information does not need to be obtained by intercepting the rendering resources. When the game is running, the electronic device will automatically save the rendering load information in the system library. Therefore, the rendering load information corresponding to each lighting scene can be directly obtained from the system library.

[0227] S502. Determine the light source distance S between each light source and the camera in the illumination scene based on the spatial coordinates of the light source and the spatial coordinates of the camera.

[0228] Specifically, the difference between the spatial coordinates of each light source and the spatial coordinates of the camera in the same illumination scene can be calculated to obtain the light source distance S between each light source and the camera in this illumination scene.

[0229] Similarly, the light source distance S between each light source and the camera can be calculated for all illumination scenes.

[0230] S503. Determine the corresponding relationship between the light source type, the light source distance S, and the rendering weight by statistically analyzing the influence of the light source type and the light source distance S on the rendering load.

[0231] Among them, for the same type of light source, when the light source distance S between it and the camera is different, the corresponding rendering load may be different. For different light sources, when the light source distance S between them and the camera is the same, the corresponding rendering load may also be different.

[0232] Therefore, the embodiments of the present application can analyze the influence of each light source on the rendering load when the light source distance S between the light source and the camera is different based on the corresponding relationship between the light source distance S between each light source and the camera and the rendering load information, so as to determine the rendering weight corresponding to each light source at different light source distances S.

[0233] That is to say, in the embodiments of the present application, the rendering weight refers to the rendering weight value corresponding to each light source type at different light source distances S.

[0234] Exemplarily, as shown in Table 2, taking the light source types including point light source, parallel light, and spotlight as an example, for the same type of light source, when the light source distance S between it and the camera is different, the corresponding rendering weight is different. For different types of light sources, when the light source distance S between them and the camera is different or the same, the corresponding rendering weight may also be different. For example, when the light source distance S between the point light source and the camera satisfies S1 < S ≤ S2, the rendering weight corresponding to this point light source is W1. When the light source distance S between the point light source and the camera satisfies S2 < S ≤ S3, the rendering weight corresponding to this point light source is W2. When the light source distance S between the point light source and the camera satisfies S3 < S ≤ S4, the rendering weight corresponding to this point light source is W3. Among them, S1 < S2 < S3 < S4. For another example, when the light source distance S between the parallel light and the camera satisfies S1 < S ≤ S2, the rendering weight corresponding to this point light source is W4. When the light source distance S between the spotlight and the camera satisfies S1 < S ≤ S2, the rendering weight corresponding to this point light source is W7.

[0235] Table 2 Corresponding relationship between light source type, distance S, and rendering weight

[0236]

[0237]

[0238] In some embodiments, a light source weight model can be obtained based on the correspondence between the light source type, the light source distance S, and the rendering weight. The input of the light source weight model can be the type of the light source corresponding to the illumination scene, the number of light sources, and the light source distance S between each light source and the camera, and the output of the light source weight model can be the rendering load corresponding to the illumination scene. Among them, the rendering weight can be used as a constant of the light source weight model. In this way, the light source weight model can calculate the rendering load corresponding to the illumination scene based on the type of the light source, the number of light sources, the light source distance S between each light source and the camera, and the rendering weight in the illumination scene.

[0239] The method for identifying the complexity of the scene light by using the basic parameters (determining the rendering weight corresponding to each light source) or the light source weight model will be described below. Specifically, please refer to the descriptions in steps S504 to S506.

[0240] S504, Intercept the rendering data of the current image frame to obtain the number of light sources, the type of light source, the spatial coordinates of the light source, and the spatial coordinates of the camera corresponding to the current image frame.

[0241] S505, Determine the light source distance between each light source and the camera in the current image frame.

[0242] S506, Based on the light source type, the light source distance of each light source in the current image frame, and the correspondence between the light source type, the light source distance S, and the rendering weight obtained in step S503, determine the rendering weight corresponding to each light source.

[0243] S507, Based on the rendering weight of each light source in the current image frame, determine the rendering load of the current image frame.

[0244] In some embodiments, the number of light sources, the type of light source, and the light source distance between each light source and the camera corresponding to the current image frame can be input into the light source weight model. In this way, the light source weight model can calculate the rendering load corresponding to the current image frame.

[0245] Exemplarily, the light source weight model can calculate the rendering load corresponding to each light source in the current image frame. Then, the rendering loads corresponding to all light sources are summed or averaged to obtain the rendering load corresponding to the current image frame.

[0246] S508, Based on the rendering load corresponding to the current image frame, determine the light complexity of the current image frame.

[0247] Exemplarily, the rendering load can be pre-divided into light complexity levels. For example, the greater the rendering load, the higher the corresponding light complexity level.

[0248] In this way, based on the pre-divided light complexity levels, the light complexity corresponding to the current image frame can be determined.

[0249] Among them, for an image frame with a higher light complexity level, the rendering load during its rendering process is heavier.

[0250] Based on the above description, as Figure 5 shown, the screen zoom recognition unit in the scene recognition module provided by the embodiments of the present application can recognize that the zoom type corresponding to the current image frame is stable screen zoom or sudden screen zoom. The vehicle recognition unit can recognize the type of vehicle driven or ridden by the main character in the current image frame. The game scene switching recognition unit can recognize the game scene corresponding to the current image frame. The scene light complexity recognition unit can recognize the light complexity corresponding to the current image frame.

[0251] In this way, the scene recognition module provided by the embodiments of the present application can, based on the rendering data corresponding to the intercepted current image frame, recognize the situation such as the screen zoom type, vehicle type, game scene, light complexity, etc. corresponding to the current image frame before rendering the current image frame. In this way, the subsequent optimization algorithm can execute the best optimization strategy based on the above-mentioned scene situations recognized in advance, improving the user experience.

[0252] In some embodiments, as Figure 13 shown, the services and processes corresponding to each optimization algorithm can subscribe to the corresponding recognition services from the scene recognition service. For example, each optimization algorithm can subscribe to the recognition services of any one or more of the four recognition units. In this way, each recognition unit in the scene recognition can send the recognition results to the scene recognition service. Then, the scene recognition service, based on the subscription situation of each optimization algorithm, sends the corresponding recognition results to each optimization algorithm.

[0253] Taking the optimization algorithms including the system scheduling algorithm, frame interpolation algorithm, super-resolution algorithm, post-processing algorithm, and screen adjustment algorithm as an example, the method for each optimization algorithm to execute the best optimization strategy based on the recognition results of the scene recognition module will be described below.

[0254] In the embodiments of the present application, the system scheduling algorithm can dynamically adjust the corresponding resource supply situation based on the rendering complexity of each image frame.

[0255] Among them, different vehicle types, game scenes, and scene light complexities can all affect the rendering complexity of performing a rendering operation on the image.

[0256] Exemplarily, the system scheduling algorithm can subscribe to the recognition services provided by the vehicle recognition unit, the game scene switching recognition unit, and the scene light complexity recognition unit in the scene recognition module. In this way, the system scheduling algorithm can adjust the corresponding resource supply situation based on the vehicle type, game scene, and light complexity before performing the rendering process. For example, in the case where it is recognized that the current image frame includes a vehicle and / or the light complexity of the current lighting scene is high, operations such as binding cores and increasing frequencies for the game thread can be performed in advance. In this way, when the rendering operation is actually performed, based on the operations such as binding cores and increasing frequencies set in advance, the running speed of high-load game scenes can be optimized.

[0257] In the embodiments of this application, the frame interpolation algorithm can draw a rendering prediction frame based on the current image frame when the game scenes of two adjacent image frames are the same and there is no sudden change in image scaling; otherwise, the rendering prediction frame is not drawn.

[0258] Exemplarily, the frame interpolation algorithm can subscribe to the recognition services provided by the image scaling recognition unit and the game scene switching recognition unit. In this way, the frame interpolation algorithm can learn in advance the difference between the current image frame and the previous image frame before performing the rendering process. Furthermore, the frame interpolation algorithm can decide whether to perform frame interpolation processing before performing the rendering process. In this way, when the rendering operation is actually performed, frame interpolation processing can be performed or not based on the frame interpolation processing decision. For example, in the case where the image scaling image type sent by the image scaling recognition unit is a sudden change in image scaling, the frame interpolation algorithm can decide not to perform frame interpolation processing on the current image frame to avoid obtaining a prediction frame with poor image quality due to a large difference between the two image frames. In this way, when the rendering operation is actually performed, the rendering prediction frame will not be drawn, thus saving the computing power overhead of the frame interpolation algorithm.

[0259] In the embodiments of this application, the super-resolution algorithm can perform image super-resolution processing on the current image frame when the game scenes of multiple consecutive image frames are the same and there is no sudden change in image scaling; otherwise, the image super-resolution processing is not performed.

[0260] Exemplarily, the super-resolution algorithm can also subscribe to the recognition services provided by the image scaling recognition unit and the game scene switching recognition unit. In this way, the super-resolution algorithm can learn in advance the difference between the current image frame and the previous image frame before performing the rendering process. For example, if the game scene sent by the game scene switching recognition unit changes, the game super-resolution algorithm can decide not to perform game super-resolution processing on the current image frame to avoid obtaining an image with poor image quality due to a large difference between the two image frames. In this way, the computing power overhead of the game super-resolution algorithm can also be saved.

[0261] In the embodiments of the present application, the post - processing algorithm includes processing for enhancing the visual effect of an image. For example, through image enhancement processing, visual effects such as the contrast, brightness, and sharpness of the image are improved. For another example, image restoration processing such as denoising, de - blurring, and occlusion repair is performed on the image.

[0262] Exemplarily, the post - processing algorithm can subscribe to the recognition services provided by the game scene switching recognition unit and the scene light complexity recognition unit. In this way, in the case of high scene light complexity, a post - processing algorithm with a smaller load overhead can be adopted; in the case of low scene light complexity, a post - processing algorithm with a larger load overhead can be adopted. In this way, corresponding post - processing algorithms can be adopted based on the scene light complexity to balance the load of the entire electronic device. For another example, the post - processing algorithm can dynamically adjust the screen brightness based on the game scene recognized by the game scene switching recognition unit. For example, if the game scene switching recognition unit recognizes that the game scene corresponding to the current image frame is a darker underground scene, the screen brightness can be increased through the post - processing algorithm to provide a better gaming experience for the user.

[0263] In the embodiments of the present application, the screen adjustment algorithm can dynamically adjust the screen refresh rate, brightness, etc. based on the game scene.

[0264] Exemplarily, the screen adjustment algorithm can subscribe to the recognition services provided by the game scene switching recognition unit. In this way, the screen adjustment algorithm adjusts the screen refresh rate, brightness, etc. based on the game scene recognized by the game scene switching recognition unit. For example, for game scenes with low requirements for follow - up performance, the screen refresh rate can be reduced; for game scenes with high requirements for follow - up performance, the screen refresh rate can be increased. For another example, for a darker underground scene, the screen brightness can be increased through the screen adjustment algorithm to provide a better gaming experience for the user.

[0265] Figure 14 This is a flowchart of a scene recognition method based on rendering data provided by the embodiments of the present application. As Figure 14 shown, the following steps can be included:

[0266] S601, before performing a rendering operation on the first image frame (which can also be referred to as the current image frame), obtain the real - time rendering data of the first image frame.

[0267] In one implementable manner, to obtain the real - time rendering data of the first image frame, the following method can be adopted: obtain the real - time rendering instruction; when the rendering instruction includes a preset rendering instruction feature, obtain the rendering data corresponding to the rendering instruction.

[0268] The real - time rendering data can include the spatial coordinates of game characters, the spatial coordinates of cameras, the spatial coordinates of vehicles, the spatial coordinates of static objects, light source information, etc.

[0269] Among them, the method for obtaining the real-time rendering data of the first image frame can be referred to the description of Figure 4 and will not be elaborated here.

[0270] S602. Based on the real-time rendering data, identify at least one of the screen scaling type, the vehicle type used by the game character, the game scene, and the scene light complexity corresponding to the first image frame.

[0271] In an implementable manner, based on the real-time rendering data, identifying the screen scaling type corresponding to the first image frame may include: based on the real-time rendering data, determining the first distance between the game character (which can also be referred to as the main character) corresponding to the first image frame and the camera; obtaining a second distance, where the second distance is the distance between the game character corresponding to the second image frame and the camera, and the second image frame is the previous image frame of the first image frame; when the difference between the first distance and the second distance (which can also be referred to as the change amount A) is greater than the distance threshold, determining that the screen scaling type corresponding to the first image frame is a screen scaling mutation; when the difference between the first distance and the second distance is less than or equal to the distance threshold, determining that the screen scaling type corresponding to the first image frame is a screen stable scaling.

[0272] In an implementable manner, based on the real-time rendering data, determining the first distance between the game character corresponding to the first image frame and the camera includes: based on the real-time rendering data, obtaining the spatial coordinates of the game character corresponding to the first image frame and the spatial coordinates of the camera; based on the spatial coordinates of the game character corresponding to the first image frame and the first offset (which can also be referred to as the offset offset1), determining the spatial coordinates of the center of the sphere around which the camera rotates with respect to the game character (equivalent to the spatial coordinates C2 of the center of the sphere in the above embodiment); the first offset is the distance offset of the spatial coordinates of the game character relative to the spatial coordinates of the center of the sphere corresponding to the camera; based on the spatial coordinates of the center of the sphere corresponding to the first image frame and the spatial coordinates of the camera, determining the first distance (equivalent to the sphere radius R_D determined by step S206 in the above embodiment).

[0273] Among them, for identifying the screen scaling type corresponding to the first image frame, reference can be made to the description of the screen scaling recognition unit in the above embodiment and will not be elaborated here.

[0274] In one implementable manner, based on real-time rendering data, identifying the vehicle type used by the game character corresponding to the first image frame includes: when the first distance is the same as the second distance, based on the spatial coordinates of the game character corresponding to the first image frame and the second offset set (the second offset set includes the offsets offset2 corresponding to each vehicle), determining the spatial coordinate set of the center of the sphere around which the camera rotates around the game character (equivalent to the spatial coordinates C4 of the multiple centers of the spheres determined in step S305 in the above embodiment); wherein, the second offset set includes the distance offsets of the spatial coordinates of the game character relative to the spatial coordinates of the center of the sphere corresponding to the camera when the game character uses different vehicles; based on the spatial coordinate set of the center of the sphere and the spatial coordinates of the camera, determining the alternative distance set (equivalent to the multiple sphere radii R_D determined in step S306 in the above embodiment); based on the first correspondence, determining the target distance from the alternative distance set; the first correspondence is the correspondence between the vehicle type and the reference distance (equivalent to the sphere radii R2 corresponding to each vehicle determined in the above embodiment), and the reference distance is the distance between the camera and the center of the sphere around which the camera rotates around the game character when the game character uses the vehicle; the target distance belongs to the reference distance in the first correspondence; based on the first correspondence and the target distance, determining the vehicle type used by the game character.

[0275] Among them, for identifying the vehicle type used by the game character corresponding to the first image frame, reference can be made to the description of the vehicle recognition unit in the above embodiment, which will not be elaborated here.

[0276] In one implementable manner, based on real-time rendering data, identifying the game scene corresponding to the first image frame includes: based on the real-time rendering data, obtaining the feature information of the static objects in the first image frame (for example, it can be the spatial coordinates of the static objects); based on the feature information of the static objects and the scene recognition database, determining the game scene corresponding to the first image frame; the scene recognition database includes multiple game scenes and the scene recognition features corresponding to each game scene in the multiple game scenes.

[0277] In one implementable manner, the scene recognition database can be established in the following manner: obtaining historical rendering data, where the historical rendering data includes the rendering data corresponding to multiple game scenes; based on the historical rendering data, extracting the feature information of the static objects corresponding to the multiple game scenes; filtering out the destructible feature information in the feature information of the static objects to obtain the alternative feature information; determining the scene recognition features corresponding to each game scene from the alternative feature information; based on each game scene and the scene recognition features corresponding to each game scene, establishing the scene recognition database.

[0278] Among them, for identifying the game scene corresponding to the first image frame, reference can be made to the description of the game scene switching recognition unit in the above embodiment, which will not be elaborated here.

[0279] In one implementable manner, identifying the scene light complexity corresponding to the first image frame based on real-time rendering data includes: determining the light source distance between each light source and the camera in the first image frame based on the real-time rendering data; determining the rendering weight corresponding to each light source based on the light source type, light source distance, and the second corresponding relationship of each light source in the first image frame, where the second corresponding relationship is the corresponding relationship between the light source type, light source distance, and the rendering weight; and determining the scene light complexity of the first image frame based on the rendering weight of each light source in the first image frame.

[0280] In one implementable manner, the second corresponding relationship can be obtained in the following way: obtaining historical rendering data, where the historical rendering data includes historical light source information corresponding to each lighting scene, the position information of the historical camera, and historical rendering load information; the historical light source information includes the number of historical light sources, the type of historical light sources, and the position information of the historical light sources; determining the historical light source distance between each historical light source and the historical camera based on the position information of the historical light source and the position information of the historical camera; and determining the second corresponding relationship between the light source type, light source distance, and the rendering weight based on the influence relationship between the historical light source distance and the historical rendering load information.

[0281] Among them, for identifying the scene light complexity corresponding to the first image frame, reference can be made to the description of the scene light complexity identification unit in the above embodiments, which will not be elaborated here.

[0282] S603. Determine an optimization strategy for the first image frame based on at least one of the screen scaling type corresponding to the first image frame, the vehicle type used by the game character, the game scene, and the scene light complexity. The optimization strategy includes at least one of the following: system scheduling strategy, frame interpolation strategy, image super-resolution strategy, image post-processing strategy, and screen adjustment strategy.

[0283] In one implementable manner, an optimization strategy for a first image frame is determined based on at least one of the screen scaling type corresponding to the first image frame, the vehicle type used by the game character, the game scene, and the scene light complexity, including: when the first image frame meets a first condition, determining that the optimization strategy for the first image frame includes binding the application thread corresponding to the first image frame to a core; the first condition includes at least one of the following: the game scene corresponding to the first image frame is the same as the game scene corresponding to a second image frame, the vehicle type used by the game character is a target vehicle type, the scene light complexity is greater than a light complexity threshold (for example, the level of the scene light complexity is greater than a preset light complexity level); when the first image frame meets a second condition, determining that the optimization strategy for the first image frame includes rendering the first image frame using an interpolation algorithm and / or an image super-resolution algorithm; the second condition includes at least one of the following: the screen scaling type is screen-stable scaling, the game scene corresponding to the first image frame is the same as the game scene corresponding to a second image frame, and the second image frame is the previous image frame of the first image frame; when the game scene corresponding to the first image frame is a low-light game scene, determining that the optimization strategy for the first image frame includes increasing the screen brightness of the electronic device; when the followability requirement of the game scene corresponding to the first image frame is greater than a followability threshold, determining that the optimization strategy for the first image frame includes reducing the screen refresh rate; when the followability requirement of the game scene corresponding to the first image frame is less than or equal to the followability threshold, determining that the optimization strategy for the first image frame includes increasing the screen refresh rate.

[0284] Among them, for the method of determining the optimization strategy for the first image frame based on at least one of the screen scaling type corresponding to the first image frame, the vehicle type used by the game character, the game scene, and the scene light complexity, reference can be made to the description of Figure 13 in the above embodiments, which will not be elaborated here.

[0285] S604, perform a rendering operation on the first image frame based on the optimization strategy.

[0286] In one implementable manner, the method further includes: obtaining configuration data of the game application corresponding to the first image frame; determining one or more of the following parameters based on the configuration data, where the parameters include a first offset, a second offset set, a first correspondence, a scene recognition database, and a second correspondence.

[0287] Exemplarily, the configuration data includes the game name; in this way, one or more parameters can be determined based on the game name.

[0288] Each method embodiment described herein can be an independent solution or can be combined according to the internal logic, and these solutions all fall within the protection scope of this application.

[0289] It can be understood that in each of the above method embodiments, the methods and operations implemented by the electronic device can also be implemented by components (such as chips or circuits) available for the electronic device.

[0290] The above embodiments introduce the scene recognition method based on rendering data provided in this application. It can be understood that in order for the electronic device to implement the above functions, it includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed in this article, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0291] The embodiment of this application also provides a processing device, which includes at least one processor and a communication interface. The communication interface is used to provide information input and / or output for the at least one processor, and the at least one processor is used to execute the method in the above method embodiment.

[0292] It should be understood that the above processing device can be a chip. For example, see Figure 15 , Figure 15 which is a structural block diagram of a chip provided in the embodiment of this application. Figure 15 The shown chip can be a general-purpose processor or a dedicated processor. The chip 700 can include at least one processor 701. Among them, the at least one processor 701 can be used to support the technical solution corresponding to any of the above embodiments.

[0293] Optionally, the chip 700 can further include a transceiver 702, and the transceiver 702 is used to accept the control of the processor 701 and is used to support the technical solution corresponding to any of the above embodiments. Optionally, Figure 15 the shown chip 700 can further include a storage medium 703. Specifically, the transceiver 702 can be replaced by a communication interface, and the communication interface provides information input and / or output for the at least one processor 701.

[0294] It should be noted that Figure 15The chip 700 shown can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), application specific integrated circuits (ASICs), system on chips (SoCs), central processor units (CPUs), network processors (NPs), digital signal processing circuits (DSPs), micro controller units (MCUs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0295] In the implementation process, the steps of the above method can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware processor, or executed and completed by the combination of the hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0296] It should be noted that the processor in the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiments may be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It is possible to implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0297] It can be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0298] According to the method provided by the embodiments of the present application, the embodiments of the present application also provide a computer program product, which includes: a computer program or instruction. When the computer program or instruction runs on a computer, the computer is enabled to execute the method of any one of the method embodiments.

[0299] According to the method provided by the embodiments of the present application, the embodiments of the present application also provide a computer storage medium, which stores a computer program or instruction. When the computer program or instruction runs on a computer, the computer is enabled to execute the method of any one of the method embodiments.

[0300] According to the method provided by the embodiments of the present application, the embodiments of the present application also provide an electronic device, including a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device is enabled to execute the method of any one of the method embodiments.

[0301] According to the method provided by the embodiments of the present application, the embodiments of the present application also provide a chip system, which includes a processor. The processor is coupled to a memory and is used to execute the computer program or instruction stored in the memory. When the computer program or instruction is executed, all or part of the steps in the method embodiments can be implemented by the chip system. The chip system can be composed of chips or can include chips and other discrete devices.

[0302] Those of ordinary skill in the art can realize that the various illustrative logical blocks and steps described in combination with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.

[0303] If the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0304] The computer storage medium, computer program product, and electronic device provided in the embodiments of this application above are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects corresponding to the methods provided above, and will not be elaborated here.

[0305] It should be understood that in various embodiments of this application, the execution order of each step should be determined according to its function and internal logic. The size of each step number does not mean the sequence of execution, and does not limit the implementation process of the embodiment.

[0306] Each part of this specification is described in a progressive manner. For the same or similar parts between various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, computer storage medium, computer program product, and electronic device, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the descriptions in the method embodiments for the relevant parts.

[0307] Although the preferred embodiments of this application have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of this application.

[0308] The above-described embodiments of this application do not constitute a limitation on the protection scope of this application.

Claims

1. A scene recognition method based on rendering data, characterized in that The method is applied to an electronic device, and the method includes: Before performing a rendering operation on a first image frame, obtaining real-time rendering data of the first image frame; Based on the real-time rendering data, identifying at least one of a screen scaling type corresponding to the first image frame, a vehicle type used by a game character, a game scene, and a scene light complexity; Based on at least one of the screen scaling type corresponding to the first image frame, the vehicle type used by the game character, the game scene, and the scene light complexity, determining an optimization strategy for the first image frame; the optimization strategy includes at least one of the following: a system scheduling strategy, an interpolation strategy, an image super-resolution strategy, an image post-processing strategy, and a screen control strategy; Based on the optimization strategy, performing a rendering operation on the first image frame.

2. The method according to claim 1, characterized in that The identifying the screen scaling type corresponding to the first image frame based on the real-time rendering data includes: Based on the real-time rendering data, determining a first distance between the game character corresponding to the first image frame and a camera; Obtaining a second distance, where the second distance is the distance between the game character corresponding to a second image frame and the camera, and the second image frame is the previous image frame of the first image frame; When the difference between the first distance and the second distance is greater than a distance threshold, determining that the screen scaling type corresponding to the first image frame is a screen scaling mutation; When the difference between the first distance and the second distance is less than or equal to the distance threshold, determining that the screen scaling type corresponding to the first image frame is a screen stable scaling.

3. The method according to claim 2, characterized in that, The determining the first distance between the game character corresponding to the first image frame and the camera based on the real-time rendering data includes: Based on the real-time rendering data, obtaining the spatial coordinates of the game character corresponding to the first image frame and the spatial coordinates of the camera; Based on the spatial coordinates of the game character corresponding to the first image frame and a first offset, determining the spatial coordinates of the center of the sphere around which the camera rotates around the game character; the first offset is the distance offset of the spatial coordinates of the game character relative to the spatial coordinates of the center of the sphere corresponding to the camera; Based on the spatial coordinates of the center of the sphere corresponding to the first image frame and the spatial coordinates of the camera, determining the first distance.

4. The method according to claim 2, characterized in that, The identifying the vehicle type used by the game character corresponding to the first image frame based on the real-time rendering data includes: When the first distance and the second distance are the same, based on the spatial coordinates of the game character corresponding to the first image frame and a second set of offsets, determining a set of spatial coordinates of the center of the sphere around which the camera rotates around the game character; wherein, the second set of offsets includes the distance offsets of the spatial coordinates of the game character relative to the spatial coordinates of the center of the sphere corresponding to the camera when the game character uses different vehicles; Based on the set of spatial coordinates of the center of the sphere and the spatial coordinates of the camera, determining a set of alternative distances; Determine a target distance from the set of alternative distances based on the first correspondence; the first correspondence is the correspondence between vehicle types and reference distances, and the reference distance is the distance between the camera and the center of the sphere around which the camera rotates when the game character uses a vehicle; the target distance belongs to the reference distances in the first correspondence; Determine the vehicle type used by the game character based on the first correspondence and the target distance.

5. The method according to claim 1, characterized in that The identifying the game scene corresponding to the first image frame based on the real-time rendering data includes: Obtain the feature information of the static objects in the first image frame based on the real-time rendering data; Determine the game scene corresponding to the first image frame based on the feature information of the static objects and the scene recognition database; the scene recognition database includes multiple game scenes and the scene recognition features corresponding to each game scene in the multiple game scenes.

6. The method according to claim 5, characterized in that The method further includes: Obtain historical rendering data, where the historical rendering data includes rendering data corresponding to multiple game scenes; Extract the feature information of the static objects corresponding to the multiple game scenes based on the historical rendering data; Filter the feature information of the static objects that can be destroyed to obtain alternative feature information; Determine the scene recognition features corresponding to each game scene from the alternative feature information; Establish a scene recognition database based on each game scene and the scene recognition features corresponding to each game scene.

7. The method according to claim 1, characterized in that, The identifying the scene light complexity corresponding to the first image frame based on the real-time rendering data includes: Determine the light source distance between each light source in the first image frame and the camera based on the real-time rendering data; Determine the rendering weight corresponding to each light source based on the light source type, light source distance, and the second correspondence of each light source in the first image frame; the second correspondence is the correspondence between light source type, light source distance, and rendering weight; Determine the scene light complexity of the first image frame based on the rendering weights of each light source in the first image frame.

8. The method according to claim 7, wherein The method further includes: Obtain historical rendering data, where the historical rendering data includes historical light source information, historical camera position information, and historical rendering load information corresponding to each lighting scene; the historical light source information includes the number of historical light sources, the types of historical light sources, and the position information of historical light sources; Determine the historical light source distance between each historical light source and the historical camera based on the position information of the historical light sources and the position information of the historical camera; Determine the second correspondence between light source type, light source distance, and rendering weight based on the influence relationship between the historical light source distance and the historical rendering load information.

9. The method according to claim 1, characterized in that The obtaining the real-time rendering data of the first image frame includes: Obtain a real-time rendering instruction; When the rendering instruction includes a preset rendering instruction feature, obtain the rendering data corresponding to the rendering instruction.

10. The method according to claim 1, wherein Determine an optimization strategy for the first image frame based on at least one of the screen scaling type corresponding to the first image frame, the vehicle type used by the game character, the game scene, and the scene light complexity, including: When the first image frame meets the first condition, determine that the optimization strategy for the first image frame includes core binding processing for the application thread corresponding to the first image frame; the first condition includes at least one of the following: the game scene corresponding to the first image frame is different from the game scene corresponding to the second image frame, the vehicle type used by the game character is the target vehicle type, the scene light complexity is greater than the light complexity threshold, and the second image frame is the previous image frame of the first image frame; When the first image frame meets the second condition, determine that the optimization strategy for the first image frame includes rendering the first image frame using an interpolation algorithm and / or an image super-resolution algorithm; the second condition includes at least one of the following: the screen scaling type is screen stable scaling, the game scene corresponding to the first image frame is the same as the game scene corresponding to the second image frame, and the second image frame is the previous image frame of the first image frame; When the game scene corresponding to the first image frame is a low-light game scene, determine that the optimization strategy for the first image frame includes increasing the screen brightness of the electronic device; When the game scene corresponding to the first image frame has a followability requirement greater than the followability threshold, determine that the optimization strategy for the first image frame includes reducing the screen refresh rate; when the game scene corresponding to the first image frame has a followability requirement less than or equal to the followability threshold, determine that the optimization strategy for the first image frame includes increasing the screen refresh rate.

11. The method according to claim 1, characterized in that The method further includes: Obtain the configuration data of the game application corresponding to the first image frame; Determine one or more of the following parameters based on the configuration data, where the parameters include a first offset, a second offset set, a first correspondence, a scene recognition database, and a second correspondence.

12. The method according to claim 11, wherein The configuration data includes a game name; determining one or more of the following parameters based on the configuration data includes: Determine one or more of the parameters based on the game name.

13. An electronic device, characterized in that, Including a memory and a processor; the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1-12.

14. A chip system, characterized in that, The chip system includes a processor; the processor is coupled to a memory, and the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the method according to any one of claims 1-12 is executed.

15. A computer-readable storage medium, characterized in that, A computer program or instruction is stored in the computer-readable storage medium. When the computer program or instruction runs on a computer, the computer executes the method according to any one of claims 1-12.

Images