Data processing method, device and equipment of virtual scene, and computer readable storage medium

Abstract

This application provides a data processing method, apparatus, device, and computer-readable storage medium for virtual scenes. The method includes: acquiring first operating environment information of the virtual scene and querying first descriptive information of the first operating environment information; decomposing an instruction to be executed to obtain at least one operation step corresponding to the instruction to be executed; executing the operation step based on the first operating environment information and the first descriptive information of the first operating environment information; and, in response to the completion of at least one operation step, performing a consistency check on the expected execution result of the instruction to be executed and the actual execution result of at least one operation step to obtain a consistency check result. This application improves the convenience and accuracy of performing operations in virtual scenes.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data processing method, apparatus, device, and computer-readable storage medium for virtual scenes. Background Technology

[0002] When users perform operations in a virtual environment, they often need to manually click, draw gestures, or perform other operations to access various components displayed in the virtual environment, and then use these components to achieve the functions they need. However, when the functions required by users need to be achieved by combining multiple components in the virtual environment, users need to perform multiple operations to achieve the desired functions, resulting in high complexity in user operations and reducing the user experience. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and computer-readable storage medium for virtual scenes, which can improve the convenience and accuracy of performing operations in virtual scenes.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method for a virtual scene, including:

[0006] Obtain the first operating environment information of the virtual scene, and query the first description information of the first operating environment information;

[0007] The instruction to be executed is decomposed to obtain at least one operation step corresponding to the instruction to be executed;

[0008] Based on the first operating environment information and the first descriptive information of the first operating environment information, the operation steps are executed;

[0009] In response to the completion of the at least one operation step, a consistency check is performed on the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step to obtain a consistency check result.

[0010] This application provides a data processing device for a virtual scene, including:

[0011] The acquisition module is used to acquire the first operating environment information of the virtual scene and query the first description information of the first operating environment information;

[0012] The decomposition module is used to decompose the instruction to be executed to obtain at least one operation step corresponding to the instruction to be executed.

[0013] The execution module is configured to execute the operation steps based on the first operating environment information and the first descriptive information of the first operating environment information;

[0014] The detection module is used to perform consistency detection on the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step in response to the completion of the operation step, and to obtain a consistency detection result.

[0015] In some embodiments, the acquisition module is further configured to acquire the component type and component coordinates of the components within the virtual scene; perform position verification processing on the component coordinates of the components within the virtual scene to obtain verified component coordinates; and combine the component type of the components and the verified component coordinates of the components to form the first operating environment information.

[0016] In some embodiments, the acquisition module is further configured to determine a first coordinate and a second coordinate indicating the two ends of the diagonal of the component in the component coordinates, wherein the diagonal endpoint corresponding to the first coordinate is located to the upper left of the diagonal endpoint corresponding to the second coordinate; when any of the following conditions are met, it is determined that the component corresponding to the component coordinate exceeds the range of the virtual scene, and the component is removed from the virtual scene: the abscissa of the first coordinate is greater than the abscissa of the second coordinate; the ordinate of the first coordinate is greater than the ordinate of the second coordinate; the abscissa of the first coordinate is less than the minimum abscissa of the scene coordinates; the ordinate of the first coordinate is less than the minimum ordinate of the scene coordinates; the abscissa of the second coordinate is greater than the maximum abscissa of the scene coordinates; the ordinate of the second coordinate is greater than the maximum ordinate of the scene coordinates; and the remaining components in the virtual scene after removal are subjected to position verification processing to obtain the verified component coordinates.

[0017] In some embodiments, the acquisition module is further configured to perform the following processing for each remaining component: when the component is located in a sliding control in the virtual scene, and the horizontal coordinate of the first coordinate is less than the horizontal coordinate of the third coordinate, the horizontal coordinate of the first coordinate is updated based on the horizontal coordinate of the third coordinate to obtain a verified first coordinate; when the component is located in a sliding control in the virtual scene, and the vertical coordinate of the first coordinate is less than the vertical coordinate of the third coordinate, the vertical coordinate of the first coordinate is updated based on the vertical coordinate of the third coordinate to obtain a verified first coordinate; when the component is located in a sliding control in the virtual scene, and the horizontal coordinate of the second coordinate is greater than the horizontal coordinate of the fourth coordinate, the horizontal coordinate of the second coordinate is updated based on the horizontal coordinate of the fourth coordinate to obtain a verified second coordinate; when the component is located in a sliding control in the virtual scene, and the vertical coordinate of the second coordinate is greater than the vertical coordinate of the fourth coordinate, the vertical coordinate of the second coordinate is updated based on the vertical coordinate of the fourth coordinate to obtain a verified second coordinate.

[0018] In some embodiments, the acquisition module is further configured to perform page pop-up window recognition processing on the virtual scene to obtain a pop-up window recognition result; when the pop-up window recognition result indicates that a page pop-up window exists in the virtual scene, the module acquires the first operating environment information of the page pop-up window in the virtual scene.

[0019] In some embodiments, the acquisition module is further configured to acquire the component coordinates of the components in the first operating environment information; perform feature extraction processing on the component coordinates to obtain the position features of the components; and retrieve the first descriptive information of the first operating environment information from the database based on the position features of the components.

[0020] In some embodiments, the execution module is further configured to determine the target component involved in the operation steps within the virtual scene; determine the component information of the target component and the first description information of the target component from the first operating environment information of the virtual scene and the first description information of the first operating environment information; and, based on the component information of the target component and the first description information of the target component, invoke an operation tool to perform the operation steps on the target component.

[0021] In some embodiments, the detection module is further configured to obtain the expected image corresponding to the expected execution result in the virtual scene and the actual image corresponding to the actual execution result in the virtual scene; extract a first image feature of the expected image and a second image feature of the actual image; and use the feature similarity between the first image feature and the second image feature as the consistency detection result of the consistency detection.

[0022] In some embodiments, the execution module is further configured to: when the execution result of the operation step is correct, obtain the execution progress of the instruction to be executed, and continue to execute subsequent operation steps of the operation step based on the execution progress; when the execution result of the operation step is incorrect, obtain the second operation environment information of the virtual scene, and query the second description information of the second operation environment information; wherein, the second operation environment information is the real-time operation environment information of the virtual scene after the operation step is executed; determine the corrected operation step corresponding to the operation step based on the execution result; and execute the corrected operation step corresponding to the operation step based on the second operation environment information and the second description information of the second operation environment information.

[0023] In some embodiments, the detection module is further configured to: obtain third operating environment information of the virtual scene and query third explanatory information of the third operating environment information when the consistency detection result indicates that the expected execution result is inconsistent with the actual execution result; wherein the third operating environment information is real-time operating environment information of the virtual scene after executing the at least one operation step; determine the correction operation steps corresponding to the instruction to be executed based on the difference between the actual execution result and the expected execution result; and execute the correction operation steps corresponding to the instruction to be executed based on the third operating environment information and the third explanatory information of the third operating environment information.

[0024] This application provides an electronic device, including:

[0025] Memory, used to store executable instructions;

[0026] The processor, when executing executable instructions stored in the memory, implements the data processing method for the virtual scene provided in the embodiments of this application.

[0027] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the data processing method for a virtual scene provided in this application.

[0028] The embodiments of this application have the following beneficial effects:

[0029] The system acquires the first operating environment information of the virtual scene and queries the first description information of the first operating environment information. It then decomposes the instruction to be executed into at least one operation step corresponding to the instruction. By decomposing the instruction, it breaks it down into at least one operation step, which is then executed sequentially to complete the instruction. If an error occurs in a certain operation step, the impact of the error can usually be limited to that step and will not affect the entire instruction. Furthermore, when a problem occurs, it can more quickly and accurately locate which specific step has an error, reducing the cost of error correction. Based on the first operating environment information and the first description information of the first operating environment information, the operation steps are executed. In response to the completion of the at least one operation step, a consistency check is performed on the expected execution result of the instruction and the actual execution result of the at least one operation step to obtain a consistency check result. By checking the consistency between the actual execution result and the expected execution result, it is ensured that the final result meets the user's needs, guaranteeing the accuracy of the execution operation. Simultaneously, by automatically executing the steps decomposed from the instruction using the operating environment information in the virtual scene, it avoids the need for manual execution by the user, improving the convenience of performing operations in the virtual scene. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the architecture of the virtual scene data processing system 100 provided in the embodiments of this application;

[0031] Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application;

[0032] Figure 3A This is a first flowchart illustrating the data processing method for a virtual scene provided in an embodiment of this application;

[0033] Figure 3B This is a schematic diagram of the second process of the data processing method for virtual scenes provided in the embodiments of this application.

[0034] Figure 4 This is a schematic diagram of the virtual scene provided in the embodiments of this application;

[0035] Figure 5 This is a schematic diagram of the first operating environment information provided in the embodiments of this application;

[0036] Figure 6 This is a schematic diagram of a pop-up window provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of a page pop-up window in a virtual scene provided in the embodiments of this application;

[0038] Figure 8 This is a schematic diagram of the data processing results provided in the embodiments of this application;

[0039] Figure 9 This is a schematic diagram of the first process for obtaining information provided in an embodiment of this application;

[0040] Figure 10 This is a schematic diagram of the second process for obtaining information provided in the embodiments of this application;

[0041] Figure 11 This is a schematic diagram of the component images provided in the embodiments of this application;

[0042] Figure 12 This is a schematic diagram of the annotation data of the components provided in the embodiments of this application;

[0043] Figure 13 This is a schematic diagram of the component retrieval process provided in the embodiments of this application;

[0044] Figure 14 This is a schematic diagram of the search results provided in an embodiment of this application;

[0045] Figure 15 This is a schematic diagram of the data processing flow provided in the embodiments of this application;

[0046] Figure 16 This is a schematic diagram of the collaborative process of the intelligent agents provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0051] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0052] 1) Virtual Scene: The virtual scene can be an interface comprising multiple components. These components may include: a personal information component displaying the player's game nickname, avatar, level, experience points, honorary titles, etc.; a virtual resource component showcasing various virtual resources within the game; a friend component providing functions such as adding friends, viewing the friend list, and interacting with friends; a chat component supporting real-time communication, including world chat, team chat, and guild chat; a leaderboard component displaying various player rankings in the game, such as level leaderboards and combat power leaderboards; a room creation component allowing players to create game rooms, set room rules, and invite friends to join; a quest component displaying various quest information, including main quests, side quests, and daily quests; and an event component displaying limited-time in-game events, such as holiday events and anniversary events.

[0053] 2) Components: Components are independent elements or modules that constitute a virtual scene and implement specific functions. Components are relatively independent and reusable parts of a complex system, divided according to function or logic. Each component has its own independent function and structure, and can operate and be maintained independently to a certain extent. For example, the shop component in a game has its own independent data processing and display logic, separate from other components in the game. Even if other components have problems, the shop component may still function normally. Components can be reused in different scenes or projects. Taking button components as an example, buttons with the same style and function may be used in multiple places such as the game's main menu, settings interface, and confirmation dialog box. Developers only need to create a button component once, which can be reused in multiple interfaces, improving development efficiency. Multiple components can be combined to form a more complex functional module or a complete user interface. For example, a virtual scene can be composed of multiple components such as personal information components, task components, and shop components. Multiple components work together to provide users with a rich interactive experience.

[0054] 3) Intelligent Agent: An intelligent agent is a widely used concept in computer science and artificial intelligence. It refers to an entity capable of autonomously perceiving, making decisions, and taking actions in a specific environment to achieve a specific goal. Intelligent agents can autonomously control their own behavior and internal state without direct external intervention, making decisions based on their own knowledge and experience. For example, a robotic vacuum cleaner can autonomously plan its cleaning path, avoid obstacles, and complete the cleaning task based on its built-in programs and sensors, without requiring user input for each step. Intelligent agents not only passively react to environmental changes but can also proactively initiate actions to achieve their own goals. For example, an investment intelligent agent with learning capabilities will proactively analyze various data in the financial market and adjust its investment portfolio according to preset investment objectives to maximize returns. Software intelligent agents typically exist in the form of programs in computer systems. For example, intelligent web crawlers in web search engines are a type of software intelligent agent that can automatically crawl the internet, capture web page information, and bring this information back to the search engine for indexing and processing. Physical intelligent agents refer to intelligent agents with physical entities, such as robots and self-driving cars. These intelligent agents perceive the physical world through various sensors and perform tasks in the physical environment through actuators. For example, industrial robots can autonomously complete tasks such as welding and assembly on factory production lines, improving production efficiency and quality.

[0055] This application provides a data processing method, apparatus, device, and computer-readable storage medium for virtual scenes, which can improve the convenience and accuracy of performing operations in virtual scenes. The following describes exemplary applications of the data processing device for virtual scenes provided in this application. The device provided in this application can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server. The following describes exemplary applications when the device is implemented as a terminal.

[0056] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the virtual scene data processing system 100 provided in the embodiments of this application. In order to support a data processing application for a virtual scene, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] Terminal 400 is used to obtain the first operating environment information and the first description information of the scene, and then transmit the first operating environment information and the first description information to server 200.

[0058] Server 200 is used to decompose the instruction to be executed to obtain at least one operation step, then execute the operation step based on the first operating environment information and the first description information, and perform a consistency check between the expected execution result and the actual execution result after the operation step is executed.

[0059] The data processing method for virtual scenes provided in this application can be applied to scenarios where operations are performed in virtual scenes, such as game scenarios, shopping scenarios, online communication scenarios, office scenarios, financial scenarios, and educational scenarios. Examples are given below:

[0060] 1) Game Scenario: In the game scenario, terminal 400 can obtain the operation environment information and instruction information of the game interface and transmit them to server 200. Server 200 can decompose the user's pending instructions (such as creating a game room, inviting game friends, purchasing virtual characters, etc.) into at least one operation step, and then execute at least one operation step based on the operation environment information and instruction information of the game interface. After the operation step is completed, the consistency between the expected execution result and the actual execution result is checked.

[0061] 2) Shopping scenario: In the shopping scenario, terminal 400 can obtain the operation environment information and instruction information of the shopping interface and transmit them to server 200. Server 200 can decompose the user's instructions to be executed (such as purchasing fixed types of goods, filtering fixed types of goods, etc.) into at least one operation step, and then execute at least one operation step based on the operation environment information and instruction information of the shopping interface. After the operation step is completed, the consistency between the expected execution result and the actual execution result is checked.

[0062] 3) Online communication scenario: In the online communication scenario, terminal 400 can obtain the operation environment information and instruction information of the communication software interface and transmit them to server 200. Server 200 can decompose the user's pending instructions (such as opening a friend's chat box, sending information to a friend via voice, etc.) into at least one operation step, and then execute at least one operation step based on the operation environment information and instruction information of the communication software interface, and perform consistency detection between the expected execution result and the actual execution result after the operation step is completed.

[0063] 4) Office Scenario: In an office scenario, terminal 400 can obtain the operating environment information and instruction information of the office software interface and transmit it to server 200. Server 200 can decompose the user's pending instructions (such as automatically replacing specific keywords in a large number of Word documents with other content, or automatically generating contracts, reports and other documents based on templates) into at least one operation step. Then, based on the operating environment information and instruction information of the communication software interface, at least one operation step is executed, and after the operation step is completed, the consistency between the expected execution result and the actual execution result is checked.

[0064] 5) Financial Scenario: In a financial scenario, terminal 400 can obtain the operating environment information and instruction information of the financial software interface and transmit it to server 200. Server 200 can decompose the user's pending instructions (such as automatically executing stock, futures, foreign exchange, and other trading operations) into at least one operation step, and then execute at least one operation step based on the operating environment information and instruction information of the communication software interface. After the operation step is completed, the consistency between the expected execution result and the actual execution result is checked.

[0065] 6) Educational Scenario: In an educational scenario, terminal 400 can obtain the operating environment information and instruction information of the educational software interface and transmit them to server 200. Server 200 can decompose the user's instructions to be executed (such as instructions for automatic question generation, test paper preparation, invigilation, and scoring) into at least one operation step. Then, based on the operating environment information and instruction information of the communication software interface, it executes at least one operation step and performs a consistency check between the expected execution result and the actual execution result after the operation step is completed.

[0066] 7) Customer service scenario: In the customer service scenario, terminal 400 can obtain the operation environment information and instruction information of the customer service communication software interface and transmit it to server 200. Server 200 can decompose the user's pending instructions (such as identifying the customer's question intent, finding the corresponding answer from the knowledge base and replying in a timely manner, etc.) into at least one operation step. Then, based on the operation environment information and instruction information of the communication software interface, at least one operation step is executed, and after the operation step is completed, the consistency between the expected execution result and the actual execution result is checked.

[0067] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the invention.

[0068] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application, with the electronic device 500 as an example. Figure 1 Taking terminal 400 as an example, Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0069] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0070] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0071] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0072] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0073] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0074] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0075] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0076] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0077] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0078] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A data processing device 555 for a virtual scene stored in memory 550 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 5551, a decomposition module 5552, an execution module 5553, and a detection module 5554. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0079] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method of the virtual scene provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0080] The following describes the data processing method for virtual scenes provided in the embodiments of this application. As mentioned above, the electronic device implementing the data processing method for virtual scenes in the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0081] See Figure 3A , Figure 3A This is a first flowchart illustrating the data processing method for a virtual scene provided in this application embodiment, which will be combined with... Figure 3A The steps shown are explained.

[0082] In step 101, the first operating environment information of the virtual scene is obtained, and the first description information of the first operating environment information is queried.

[0083] In practical implementation, a virtual scene can be an interface comprising multiple components. Taking a virtual scene in a game as an example, the components may include: a personal information component to display the player's game nickname, avatar, level, experience points, honorary titles, etc.; a virtual resource component to present various virtual resources within the game; a friend component to provide functions such as adding friends, viewing the friend list, and interacting with friends; a chat component to support real-time communication between users, including world chat, team chat, and guild chat; a leaderboard component to display various player rankings in the game, such as level leaderboards and combat power leaderboards; a room creation component to allow players to create game rooms, set room rules, and invite friends to join; a quest component to display various quest information, including main quests, side quests, and daily quests; and an event component to display limited-time events within the game, such as holiday events and anniversary events.

[0084] In practical implementation, the first operating environment information can be the attributes, coordinates, and functions of components in the virtual scene. Component attributes can include information such as the component's access path and name. Component coordinates can be the component's coordinates in the virtual scene. Component functions can be the specific operations and behaviors that the component can perform in the virtual scene. Taking a game scene as an example, a virtual attack item component has functions such as attack and damage calculation. Users can attack by operating the virtual attack item component. Different attack effects are produced according to the different attributes and skills of the virtual attack item. Another example is that a portal component has the function of quickly moving a virtual character from one location to another. In addition, some functional components can also be linked with other components to realize more complex interaction logic. For example, a sensor component can detect changes in the surrounding environment and trigger corresponding events based on the detection results, such as controlling the switching of lights or activating mechanisms.

[0085] In some embodiments, obtaining the first operating environment information of the virtual scene in step 101 can be achieved through methods such as... Figure 3B Steps 1011 to 1013 shown are implemented as follows:

[0086] In step 1011, the component type and component coordinates of the components in the virtual scene are obtained.

[0087] In practice, the way to obtain the component type and component coordinates of a component is to first identify the components included in the virtual scene, and then look up the component type and component coordinates of each component in the database through the component's access path. Components can be displayed in various forms, and the component type can include text components, icon components, and video components, etc.

[0088] In step 1012, the component coordinates of the components in the virtual scene are subjected to position verification processing to obtain the verified component coordinates.

[0089] In actual implementation, due to the limitations of the terminal's display, only a portion of the virtual scene can be displayed at a time, and similarly, only a portion of the components can be displayed. Furthermore, some components may be split into two parts by the display and cannot be fully displayed. In response to the above situations, errors may occur when selecting components. Therefore, it is necessary to verify the component coordinates of the components within the virtual scene.

[0090] In some embodiments, the position verification processing of the component coordinates of the components in the virtual scene in step 1012 to obtain the verified component coordinates can be implemented by the following technical solution: determining the first coordinate and the second coordinate indicating the two ends of the diagonal of the component in the component coordinates, wherein the diagonal endpoint corresponding to the first coordinate is located to the upper left of the diagonal endpoint corresponding to the second coordinate; when any one of the following conditions is met, determining that the component corresponding to the component coordinate exceeds the range of the virtual scene, and removing the component from the virtual scene: the abscissa of the first coordinate is greater than the abscissa of the second coordinate; the ordinate of the first coordinate is greater than the ordinate of the second coordinate; the abscissa of the first coordinate is less than the minimum abscissa of the scene coordinates; the ordinate of the first coordinate is less than the minimum ordinate of the scene coordinates; the abscissa of the second coordinate is greater than the maximum abscissa of the scene coordinates; the ordinate of the second coordinate is greater than the maximum ordinate of the scene coordinates; and performing position verification processing on the remaining components in the virtual scene after removal to obtain the verified component coordinates.

[0091] In practice, the first and second coordinates of the two ends of the diagonal of the component can be determined from the component coordinates. For example, if the outer shape of the component is a rectangle, the coordinates of the top left corner of the outer rectangle can be used as the first coordinate, and the coordinates of the bottom right corner of the outer rectangle can be used as the second coordinate. As another example, if the outer shape of the component is a hexagon, the coordinates of the top left corner of the diagonal from the top left corner to the bottom right corner of the hexagon can be used as the first coordinate, and the coordinates of the bottom right corner of the diagonal can be used as the second coordinate.

[0092] The following explains the situations where components cannot be displayed or used correctly. When the x-coordinate of the first coordinate is greater than the x-coordinate of the second coordinate, or the y-coordinate of the first coordinate is less than the y-coordinate of the second coordinate, it indicates that the current component's coordinates are incorrectly identified, and the current component cannot be displayed correctly. That is, when the x-coordinate of the top-left corner of the diagonal (the point corresponding to the first coordinate) is greater than the x-coordinate of the bottom-right corner of the diagonal (the point corresponding to the second coordinate), it means that the endpoint of the top-left corner of the diagonal is to the right of the endpoint of the bottom-right corner of the diagonal, indicating that the current component's coordinates are incorrectly identified, and the component needs to be removed. Similarly, when the y-coordinate of the top-left corner of the diagonal is greater than the y-coordinate of the bottom-right corner of the diagonal (the positive direction of the vertical axis in the coordinate system is downward), it means that the endpoint of the top-left corner of the diagonal is below the endpoint of the bottom-right corner of the diagonal, indicating that the current component's coordinates are incorrectly identified, and the component needs to be removed.

[0093] When the x-coordinate of the first coordinate is less than the minimum x-coordinate of the scene coordinates (the coordinates of points within the area displayed on the screen), it means the top-left corner of the diagonal is located to the left of the area displayed on the screen, indicating that the component exceeds the display area and needs to be removed. When the y-coordinate of the first coordinate is less than the minimum y-coordinate of the scene coordinates, it means the top-left corner of the diagonal is located above the area displayed on the screen, indicating that the component exceeds the display area and needs to be removed. When the x-coordinate of the second coordinate is greater than the maximum x-coordinate of the scene coordinates, it means the bottom-right corner of the diagonal is located to the right of the area displayed on the screen, indicating that the component exceeds the display area and needs to be removed. When the y-coordinate of the second coordinate is greater than the maximum y-coordinate of the scene coordinates, it means the bottom-right corner of the diagonal is located below the area displayed on the screen, indicating that the component exceeds the display area and needs to be removed.

[0094] By using the above method, components that are not in the display area can be removed, avoiding the need to click on components located outside the display area during subsequent operation steps, thereby improving the accuracy of operation steps.

[0095] In some embodiments, the sliding control in the virtual scene includes a third coordinate and a fourth coordinate indicating the two ends of the diagonal of the sliding control. The above-mentioned position verification processing of the remaining components in the virtual scene after removal, to obtain the verified component coordinates, can be implemented through the following technical solution: For each remaining component, perform the following processing: When the component is located in the sliding control in the virtual scene, and the horizontal coordinate of the first coordinate is less than the horizontal coordinate of the third coordinate, update the horizontal coordinate of the first coordinate based on the horizontal coordinate of the third coordinate to obtain the verified first coordinate; when the component is located in the sliding control in the virtual scene, and the vertical coordinate of the first coordinate is less than the horizontal coordinate of the third coordinate, update the horizontal coordinate of the first coordinate based on the horizontal coordinate of the third coordinate to obtain the verified first coordinate; When the coordinate is less than the ordinate of the third coordinate, the ordinate of the first coordinate is updated based on the ordinate of the third coordinate to obtain a verified first coordinate; when the component is located in a sliding control in the virtual scene, and the x-coordinate of the second coordinate is greater than the x-coordinate of the fourth coordinate, the x-coordinate of the second coordinate is updated based on the x-coordinate of the fourth coordinate to obtain a verified second coordinate; when the component is located in a sliding control in the virtual scene, and the ordinate of the second coordinate is greater than the ordinate of the fourth coordinate, the ordinate of the second coordinate is updated based on the ordinate of the fourth coordinate to obtain a verified second coordinate.

[0096] In some embodiments, when the area displayed on the screen corresponds to the area of ​​the sliding control, it may be possible that when a portion of a component is displayed on the screen, clicking on that portion of the component will enable the functionality of that component. Figure 4 As shown, Figure 4 This is a schematic diagram of a virtual scene provided in an embodiment of this application.

[0097] Depend on Figure 4 As can be seen from component 401, the portion of component 401 displayed on the screen is 402. Therefore, clicking on the area of ​​component 402 will achieve the function of component 401. During the clicking process, the center coordinates of the component are usually calculated first, and then the component is clicked by clicking the center point corresponding to those coordinates. And as... Figure 4 The center point 403 corresponding to the center coordinates of component 401 is located outside the screen. If clicking the center point corresponding to the center coordinates of component 401 cannot click component 401 normally, then the position of component 401 needs to be verified.

[0098] In practice, when the x-coordinate of the first coordinate is less than the x-coordinate of the third coordinate, it indicates that the top-left corner of the component's diagonal is located to the left of the top-left corner of the slider control's diagonal. Therefore, the x-coordinate of the first coordinate needs to be updated to the x-coordinate of the third coordinate so that the top-left corner of the updated component's diagonal is within the slider control. For example, if the x-coordinate of the first coordinate is 3 and the x-coordinate of the third coordinate is 5, then the x-coordinate of the first coordinate is updated to 5.

[0099] When the ordinate of the first coordinate is less than the ordinate of the third coordinate, it indicates that the top-left corner of the component's diagonal is above the top-left corner of the slider control's diagonal. Therefore, the ordinate of the first coordinate needs to be updated to the ordinate of the third coordinate so that the top-left corner of the updated component's diagonal is within the slider control. For example, if the ordinate of the first coordinate is 3 and the ordinate of the third coordinate is 5, then the ordinate of the first coordinate should be updated to 5.

[0100] When the x-coordinate of the second coordinate is greater than the x-coordinate of the fourth coordinate, it indicates that the bottom right corner of the component's diagonal is located to the right of the bottom right corner of the slider control's diagonal. Therefore, the x-coordinate of the second coordinate needs to be updated to the x-coordinate of the fourth coordinate so that the bottom right corner of the updated component's diagonal is within the slider control. For example, if the x-coordinate of the second coordinate is 10 and the x-coordinate of the fourth coordinate is 8, then the x-coordinate of the second coordinate should be updated to 8.

[0101] When the ordinate of the second coordinate is greater than the ordinate of the fourth coordinate, it indicates that the bottom right corner of the component's diagonal is below the bottom right corner of the slider control's diagonal. Therefore, the ordinate of the second coordinate needs to be updated to the ordinate of the fourth coordinate so that the bottom right corner of the component's diagonal is within the slider control. For example, if the ordinate of the second coordinate is 10 and the ordinate of the fourth coordinate is 8, then the ordinate of the second coordinate should be updated to 8.

[0102] In actual implementation, after the coordinate values ​​of the first and second coordinates are updated, the center point of the component can be determined based on the updated first and second coordinates. Specifically, the center point of the component can be determined by averaging the abscissa of the first coordinate and the abscissa of the second coordinate, and the resulting value is the abscissa of the component center point. Then, the ordinate of the ordinate of the first coordinate and the ordinate of the second coordinate is averaged, and the resulting value is the ordinate of the component center point.

[0103] By using the above method, the center point of some components displayed in the sliding control can be adjusted to the display area. Then, by clicking the adjusted center point of the component, the function of the component can be realized. This avoids directly removing some components displayed in the sliding control, thus improving the efficiency and accuracy of subsequent operation steps.

[0104] In step 1013, the component type of the component and the verified component coordinates of the component are combined to form the first operating environment information.

[0105] In practice, after obtaining the component type and the verified component coordinates, the component type and the verified component coordinates can be combined to obtain the first operating environment information. The specific combination method can be as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of the first operating environment information provided in the embodiments of this application.

[0106] like Figure 5 As shown, the first operating environment information includes the names of components, such as... Figure 5 The components shown are “ / Bootobj / CUIManager / Forms / Form_Bag / Center / TopCommon / Panel_Menu / pnlMenu / M” and “ / Bootobj / CUIManager / Forms / Form_Bag / Center / TopCommon / Panel_TopBg / Text”; the first operating environment information also includes the text name of the component, such as… Figure 5The "in-game performance" and "backpack" shown; the first operating environment information also includes the component's "top left corner coordinates," "bottom right corner coordinates," and "center coordinates," such as... Figure 5 The "In-Game Emoticons" component shown has the following coordinates: top left corner x: 118, y: 840; bottom right corner x: 268, y: 960; center x: 193, y: 900; and "backpack" component x: 339, y: 24; bottom right corner x: 426, y: 96; center x: 382, ​​y: 60. The first operating environment information also includes information such as whether the component can be scrolled (i.e., whether the component is located in a sliding control), whether it can be clicked, and whether it is selected. Figure 5 The "In-Game Performance" component shown is not scrollable, not clickable, and not selected; the "Backpack" component is not scrollable, not clickable, and not selected.

[0107] By verifying the component coordinates, the accuracy of the component coordinates included in the first operating environment information is improved, avoiding errors in subsequent operation steps caused by inaccurate component coordinates in the first operating environment information, and thus improving the accuracy of the operation steps.

[0108] In some embodiments, obtaining the first operating environment information of the virtual scene in step 101 can be achieved by the following technical solution: performing page pop-up window recognition processing on the virtual scene to obtain a pop-up window recognition result; when the pop-up window recognition result indicates that there is a page pop-up window in the virtual scene, obtaining the first operating environment information of the page pop-up window in the virtual scene.

[0109] In actual implementation, pop-ups may appear in virtual scenarios, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of a pop-up window provided in an embodiment of this application.

[0110] like Figure 6 As shown, pop-up window 601 appears on the displayed screen. Components in the background behind pop-up window 601 will become unclickable. Figure 6 The components "Component 1", "Component 2", "Component 3", and "Component 4" shown can only be operated on at this time, as shown in pop-up window 601. Figure 6 Components 602 and 603 are shown in the diagram.

[0111] Therefore, before obtaining the initial operational environment information in the virtual scene, it is necessary to first identify pop-up windows in the virtual scene to determine whether page pop-ups exist. Pop-up window identification in the virtual scene can be performed using a multimodal model. Specifically, firstly, screenshots of pages containing various pop-ups are collected. These screenshots should cover different types of pop-ups, such as advertising pop-ups, warning pop-ups, and prompt pop-ups, and come from different web pages and applications to ensure data diversity. Simultaneously, text information from the pages and pop-ups is extracted, including titles, content descriptions, button text, etc. Next, the collected data is labeled, clearly indicating the location of pop-ups in the images (using bounding boxes) and labeling the type of pop-ups (e.g., advertising, warning, etc.). For text data, key information related to the pop-ups is labeled. Finally, the processed data is used as training samples to train the multimodal model, which can be a You Only Look Once (YOLO) model.

[0112] In practice, the acquired virtual scene pages can be input into a trained multimodal model, which will then output the predicted page pop-ups included in the virtual scene, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of a page pop-up window in a virtual scene provided in the embodiments of this application.

[0113] Depend on Figure 7 It can be seen that the content in box 701 is the page pop-up window included in the virtual scene. Meanwhile, box 702 is the probability predicted by the multimodal model that box 701 is a page pop-up window (the probability of it being a page pop-up window is 0.98). After identifying the page pop-up window included in the virtual scene, the first operation environment information of the page pop-up window in the virtual scene can be obtained. Specifically, the component type and component coordinates of the components in the page pop-up window can be obtained first. Then, the position verification processing of the component coordinates of the components in the page pop-up window is performed to obtain the verified component coordinates. The component type and the verified component coordinates of the component are combined to form the first operation environment information of the page pop-up window.

[0114] By using the above method, the first operation environment information of the pop-up window can be identified after it appears in the virtual scene, without needing to identify the first operation environment information of the background page in the virtual scene. This improves computational efficiency and avoids errors caused by operating on components in the background page when a pop-up window exists, thus improving the accuracy of subsequent operation steps.

[0115] In some embodiments, the first descriptive information of the first operating environment information in step 101 can be implemented by the following technical solution: obtaining the component coordinates of the component in the first operating environment information; performing feature extraction processing on the component coordinates to obtain the position features of the component; and retrieving the first descriptive information of the first operating environment information from the database based on the position features of the component.

[0116] In actual implementation, in order to ensure the aesthetics of the virtual scene page, the components in the virtual scene may only be displayed through icons, animations, etc., without text content. In this case, it is necessary to extract the features of the coordinates of these components in order to determine the actual function of these components.

[0117] In actual implementation, feature extraction processing can be performed on the component coordinates of the components in the first operating environment information to obtain the component's position features. Then, based on the position features, similarity calculation is performed between the position features and the position features of each component stored in the database. The component with the highest similarity to the position feature stored in the database is taken as the component in the first operating environment information, and the function, name, access path and other information of the component recorded in the database are taken as the first descriptive information.

[0118] By using the above method, the first description information of unrecognizable icons, videos, and other components in the virtual scene can be identified, avoiding accidental touches on components in the virtual scene and improving the accuracy of subsequent operation steps.

[0119] In step 102, the instruction to be executed is decomposed to obtain at least one operation step corresponding to the instruction to be executed.

[0120] In practice, the decomposition of the instruction to be executed can be based on the execution logic of the instruction, breaking it down into at least one operation step. For example, if the instruction to be executed A is to invite a target friend to enter the room, then instruction A can be broken down into operation step A: displaying the friend list; operation step B: selecting the target friend and opening the target friend's function list; operation step C: inviting the target friend to join a team; and operation step D: closing the pop-up window on the successful invitation page.

[0121] In practical implementation, the process of decomposing instructions can be carried out by an intelligent agent. The agent outputs at least one decomposed operational step. An intelligent agent is an entity capable of sensing its environment and autonomously taking actions based on the sensed information to achieve a specific goal. It can be a software program or a hardware device. The intelligent agent can make decisions and take actions autonomously without direct human intervention. For example, in a smart home system, a smart thermostat, as an intelligent agent, can automatically adjust the air conditioner's on / off state and temperature setting based on temperature information sensed by an indoor temperature sensor.

[0122] In step 103, the operation steps are performed based on the first operating environment information and the first description information of the first operating environment information.

[0123] In actual implementation, after obtaining the first operating environment information and the first description information of the first operating environment information, the first operating environment information and the first description information of the first operating environment information can be input into the intelligent agent. The intelligent agent makes a decision to determine the operating tools to be called and the components to perform the operating steps. The intelligent agent then calls the operating tools to perform the operating steps on the target components.

[0124] In some embodiments, the first operating environment information includes component information of components within the virtual scene; the execution of the operation step in step 103 based on the first operating environment information and the first descriptive information of the first operating environment information can be implemented by the following technical solution: determining the target component involved in the operation step within the virtual scene; determining the component information of the target component and the first descriptive information of the target component from the first operating environment information of the virtual scene and the first descriptive information of the first operating environment information; and calling an operation tool to execute the operation step on the target component based on the component information of the target component and the first descriptive information of the target component.

[0125] In actual implementation, the components required to perform the operation steps are first determined. For example, if the operation step is to display the friend list, the required target component is the friend component that implements the function of displaying the friend list. If the operation step is to select the target friend and open the target friend's function list, the required target components are the target friend's character component and function component. If the operation step is to invite the target friend to join a team, the required target component is the invitation team component that implements the function of inviting friends to join a team.

[0126] In actual implementation, after the target component is determined, the component information of the target component and the first description information of the component information of the target component can be determined from the first operating environment information and the first description information. For example, if the target component is a friend component, the center coordinates of the friend component and the component type of the friend component can be determined from the first operating environment information, and the functions that the friend component can achieve and the access path can be determined from the first description information.

[0127] In practice, operation steps can be performed on the target component using operation tools. These tools can include left-click tools, right-click tools, rotation tools, and page-sliding tools. If the operation step is to display the friend list, after determining the center coordinates of the friend component (target component), the left-click tool can be used to click the position corresponding to the center coordinates of the friend component to perform the operation step of displaying the friend list.

[0128] In this way, the target component can be operated by calling the operation tool based on the first operation environment information to perform the corresponding operation steps. This achieves the goal that the operation steps can be performed without user intervention, simply by the intelligent agent calling the operation tool, thus improving the efficiency of the operation steps.

[0129] In step 104, in response to the completion of the at least one operation step, a consistency check is performed on the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step to obtain a consistency check result.

[0130] In some embodiments, the consistency detection of the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step in step 104 can be achieved by the following technical solution: obtaining the expected screen corresponding to the expected execution result in the virtual scene and the actual screen corresponding to the actual execution result in the virtual scene; extracting the first image feature of the expected screen and the second image feature of the actual screen; and using the feature similarity between the first image feature and the second image feature as the consistency detection result of the consistency detection.

[0131] In practice, before executing an operation step, the expected screen after the operation step is predicted. For example, if the operation step is to display the friend list, the predicted screen might be that the images in other areas remain unchanged, and the friend list is opened on the right side of the display area. This prediction of the expected screen after the operation step can be achieved through an intelligent agent.

[0132] In actual implementation, after the operation steps are executed, the actual image of the virtual scene after the operation steps are executed can be obtained by taking screenshots or other means. Then, image features can be extracted from the actual image and the expected image to obtain the first image features of the expected image and the second image features of the actual image. The feature similarity between the first image features and the second image features is determined. If the feature similarity exceeds the similarity threshold, the consistency detection result is determined to be qualified. If the feature similarity does not exceed the similarity threshold, the consistency detection result is determined to be unqualified.

[0133] In practice, the methods for extracting the first image features of the actual image and the second image features of the expected image can be as follows: First, the image features can be automatically learned using the convolutional layers, pooling layers, and fully connected layers of a convolutional neural network. The convolutional layers use convolutional kernels to slide across the actual and expected images, extracting local features. The pooling layers reduce the dimensionality of the feature maps, thus reducing computation. The fully connected layers classify or regress the extracted features. Alternatively, pre-trained neural network models on large-scale datasets can be used to extract image features from the actual and expected images. These pre-trained neural network models have learned rich image feature representations and can be used as feature extractors in new tasks.

[0134] In practice, the feature similarity between the first image feature and the second image feature can be determined by measuring the Euclidean distance, Manhattan distance, Chebyshev distance, cosine similarity, etc.

[0135] It should be noted that consistency checks on operation steps can be performed once after each operation step is completed, after a preset number of operation steps are completed, or after all operation steps are completed. The specific timing of consistency checks can be selected according to the actual situation.

[0136] The above method enables consistency detection by comparing the expected and actual images, improving the accuracy of consistency detection. Furthermore, since the cost of acquiring the expected and actual images is very low, the cost of consistency detection is reduced.

[0137] In some embodiments, for any of the operation steps, after executing the operation step, the following technical solution may also be performed: when the execution result of the operation step is correct, the execution progress of the instruction to be executed is obtained, and the subsequent operation steps of the operation step are continued to be executed based on the execution progress; when the execution result of the operation step is incorrect, the second operation environment information of the virtual scene is obtained, and the second description information of the second operation environment information is queried; wherein, the second operation environment information is the real-time operation environment information of the virtual scene after the operation step is executed; the corrected operation step corresponding to the operation step is determined based on the execution result; the corrected operation step corresponding to the operation step is executed based on the second operation environment information and the second description information of the second operation environment information.

[0138] In practice, when the execution result of the operation steps is correct, the progress of the instruction to be executed can be obtained so as to execute the subsequent operation steps. For example, if the operation steps corresponding to the instruction to be executed A are operation step A → operation step B → operation step C, then after confirming that the execution result of operation step A is correct, operation step B is executed.

[0139] In actual implementation, if the execution result of the operation steps is incorrect, the second operation environment information and the second description information of the virtual scene are obtained. For example, if the operation step is to display the friend list, but after executing the operation step, the mall interface is opened, then the second operation environment information and the second description information of the virtual scene that opened the mall interface need to be obtained.

[0140] In actual implementation, after determining the second operating environment information and the second instruction information, the second operating environment information and the second instruction information can be input into the intelligent agent, and the intelligent agent can make a new decision to obtain the corrected operation steps. Continuing with the previous example, if the mall interface is opened by mistake, the intelligent agent can formulate the corrected operation steps to exit the mall interface.

[0141] In practice, after determining the corrective action steps, the corresponding action tool can be invoked to perform the corrective action steps on the component based on the operating environment information and the instruction information. Continuing with the previous example, if the online store interface is opened incorrectly, the exit component for exiting the online store interface can be determined from the instruction information, and the center coordinates of the exit component can be determined from the operating environment information. The agent can then invoke the click action tool to click the center coordinates of the exit component to perform the corrective action steps.

[0142] By using the above method, if an error occurs during the execution of an operation step, the erroneous operation step can be automatically corrected by the prescribed correction operation steps, thus ensuring the accuracy of the execution result corresponding to the instruction to be executed.

[0143] In some embodiments, after performing consistency detection on the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step in step 104, the following technical solution may also be performed: when the consistency detection result indicates that the expected execution result is inconsistent with the actual execution result, the third operation environment information of the virtual scene is obtained, and the third description information of the third operation environment information is queried; wherein, the third operation environment information is the real-time operation environment information of the virtual scene after the execution of the at least one operation step; based on the difference between the actual execution result and the expected execution result, the correction operation step corresponding to the instruction to be executed is determined; based on the third operation environment information and the third description information of the third operation environment information, the correction operation step corresponding to the instruction to be executed is executed.

[0144] In actual implementation, if the consistency detection result after all the operation steps corresponding to the instruction to be executed are completed, indicating that the expected execution result is inconsistent with the actual execution result, then the third operation environment information and the third explanatory information of the current virtual scene can be obtained. For example, if the instruction to be executed is to invite friends to enter the room, and the result after all the operation steps are completed is to display information about virtual characters in the virtual scene, then the third operation environment information and the third explanatory information of the interface that displays information about virtual characters in the virtual scene can be obtained.

[0145] In practice, the third operating environment information and the third instruction information can be input into the intelligent agent. The intelligent agent will then decide on the corrective operation steps based on the difference between the actual execution result and the expected execution result. Continuing the previous example, if the instruction to be executed is to invite a friend to enter the room, and the result after executing all the operation steps is to display the information of a virtual character in the virtual scene, then the corrective operation steps can be determined as follows: Corrective operation step A: return to the main interface; Corrective operation step B: display the friend list; Corrective operation step C: select the target friend and open the target friend's function list; Corrective operation step D: invite the target friend to team up; Corrective operation step E: close the invitation success pop-up.

[0146] In practice, after determining the correction operation steps, the correction operation steps corresponding to the instruction to be executed can be executed based on the third operating environment information and the third description information of the third operating environment information. Continuing the previous example, when returning to the main interface by executing the above correction operation step A, the return component that can realize the function of returning to the main interface can be determined from the third description information, and the midpoint coordinates of the return component can be determined from the third operating environment information. By calling the click tool in the operation tool, the center coordinates of the return component are clicked to realize the return from the mall interface to the main interface.

[0147] By using the above method, even if the execution result does not match the expectation after the instruction to be executed is completed, the execution result can be corrected by determining the correction operation steps, thus ensuring the accuracy of the final execution result.

[0148] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0149] The virtual scene data processing method provided in this application embodiment can, after the user inputs the target operation steps and the content to be asserted (expected result), enable the platform to automatically collect game page information (i.e., the first operation environment information of the aforementioned virtual scene), and combine it with multiple large models to complete page operations and page assertions. For example... Figure 8 As shown, Figure 8 This is a schematic diagram of the data processing results provided in the embodiments of this application.

[0150] exist Figure 8 In box 801, the user can input the desired operation (i.e., the instruction to be executed mentioned above) in the execution scenario, such as... Figure 8 The content in box 801 indicates "Go to the personal homepage and change the displayed hero to Mai Shiranui". The assertion (expected result) in box 802 is the execution result of the instruction to be executed predicted by the agent (i.e., the expected execution result mentioned above). The log window in box 803 displays the log of the agent's execution of the instruction to be executed, such as... Figure 8The box 803 indicates "-Round 3 (representing the third operation step corresponding to the instruction to be executed) -> Start Event Decision". Model thinking: According to the user instruction, the displayed hero needs to be changed to Mai Shiranui. The current page has already opened the hero selection interface, and it is necessary to find Mai Shiranui and click to select her. Model operation: Click Mai Shiranui's hero icon. Model action: Click (1279, 403) (i.e., the coordinates of the center point of the target component), collect mobile phone environment information > Start Planning - Round 4 (representing the fourth operation step corresponding to the instruction to be executed) Start Event Decision. Model thinking: According to the historical operations, the user has already entered the personal homepage, clicked the "Change Hero" button, and clicked Mai Shiranui's hero icon. Next The next step requires confirmation on whether to save changes or stop the operation directly. Model operation: Click the "Save" button to save changes. Model action: Click (1991, 910) > Collect mobile phone environment information > Start planning, Round 5 (representing the fifth operation step corresponding to the instruction to be executed) > Start event decision. Model thinking: Based on historical operations, clicking the personal avatar to enter the personal homepage, clicking the "Change Hero" button, clicking the Shiranui Mai hero icon, and clicking the "Save" button to save changes have been completed. All user instruction requirements have been completed. Model operation: All user instruction requirements have been completed, and no further operation is required. The content in box 804 shows the actual execution result of executing the instruction to be executed.

[0151] The following is combined Figure 9 This describes the process of obtaining the first operating environment information and the first descriptive information of the virtual scene. Figure 9 This is a schematic diagram of the first process for obtaining information provided in the embodiments of this application.

[0152] In step 901, the terminal obtains the displayed screenshot, valid control information, and code layer information.

[0153] The terminal can take a screenshot of the screen currently displayed. The following section combines... Figure 10 This describes the process of determining valid control information (i.e., the first operating environment information mentioned above) and code-level information (i.e., the first description information mentioned above). Figure 10 This is a schematic diagram of the second process for obtaining information provided in the embodiments of this application.

[0154] In step 1001, the terminal traverses all components included in the virtual scene.

[0155] The terminal finds the component information of all components included in the virtual scene by traversing the scene, and determines the type of the component based on the component's suffix, i.e., whether the component is an operable component or a text component, and obtains the component coordinates of each component and the scene coordinates of the virtual scene.

[0156] In step 1002, the terminal determines whether the screen coordinates exceed the mobile phone screen range and whether the component is outside the visible range by using component coordinates and scene coordinates.

[0157] Get the coordinates of the top left corner (i.e., the first coordinates mentioned above) (leftI, topI) and the coordinates of the bottom right corner (i.e., the second coordinates mentioned above) (rightI, bottomI) of the in-game component.

[0158] Check if the component coordinates exceed the screen boundaries, that is, check if leftI is greater than rightI or topI is greater than bottomI, and if rightI and bottomI are less than 0, or if leftI and topI are greater than the screen width ({x}) and height ({y}). If they exceed the boundaries, skip the current component.

[0159] If the component is located within a sliding control, adjust the component's coordinates according to the sliding control's boundaries. Specifically:

[0160] If leftI is less than the x-coordinate of the top-left corner of the sliding control (i.e., the x-coordinate of the third coordinate mentioned above), then leftI is adjusted to the x-coordinate of the top-left corner of the sliding control; if topI is less than the y-coordinate of the top-left corner of the sliding control (i.e., the y-coordinate of the third coordinate mentioned above), then topI is adjusted to the y-coordinate of the top-left corner of the sliding control; if rightI is greater than the x-coordinate of the bottom-right corner of the sliding control (i.e., the x-coordinate of the fourth coordinate mentioned above), then rightI is adjusted to the x-coordinate of the bottom-right corner of the sliding control; if bottomI is greater than the y-coordinate of the bottom-right corner of the sliding control (i.e., the y-coordinate of the fourth coordinate mentioned above), then bottomI is adjusted to the y-coordinate of the bottom-right corner of the sliding control. These adjustments ensure that the adjusted leftI and topI are not less than 0, and that the adjusted rightI and bottomI do not exceed the screen width (x) and height (y). The adjusted component coordinates are then output to ensure that the component is displayed correctly on the screen or within the sliding page.

[0161] In step 902, the terminal performs pop-up window recognition and editable text conversion based on the screenshot.

[0162] After the terminal obtains the screenshot, it still cannot accurately identify the pop-up scene. Therefore, it is necessary to use a multimodal model to identify the pop-up scene. This solution mainly uses the YOLO model for labeled training of pop-up scene, so that the model can identify the pop-up scene in the virtual scene and provide the recognition result to the intelligent agent that makes the operation decision.

[0163] When collecting screenshots and component information from the game page, many components lacked text descriptions, and their icons were of types that the model could not recognize, such as... Figure 11 As shown, Figure 11This is a schematic diagram of the component images provided in the embodiments of this application. Figure 11 In this context, image 1101 is an image that the model cannot perceive; that is, the model cannot know the function that the current component can perform based solely on this image. Therefore, it is necessary to label the information of the corresponding component in advance, then add the labeled data to the database, and finally retrieve the relevant information from the database as the first descriptive information of the first operating environment. The labeled data is as follows: Figure 12 As shown, Figure 12 This is a schematic diagram of the annotation data of the components provided in the embodiments of this application.

[0164] exist Figure 12 In the process of annotating components, the component's name and descriptive information will be annotated, such as... Figure 12 The labeled content includes the component named " / Center / PlayerBtn / Dianquan / Button" with the description "This is a recharge point button control, related terms: points, recharge", the component named " / PlayerBtn / VipEntryRoot / ActiveRoot / Entry" with the description "This is a VIP button control, related terms: first recharge, VIP", and the component named " / Center / Friend / FriendList / ExtendBtn" with the description "This is an expand friend list control, related terms: friend list, sidebar".

[0165] The retrieval process can be as follows: Figure 13 As shown, Figure 13 This is a schematic diagram of the component retrieval process provided in the embodiments of this application.

[0166] In step 1301, the terminal obtains the search object.

[0167] After traversing the components, the terminal will use the components that cannot be recognized by the model as the search objects.

[0168] In step 1302, the terminal extracts features from the search object.

[0169] The terminal extracts features from the search object to obtain the object features of the search object.

[0170] In step 1303, the terminal retrieves objects from the database that have similar characteristics to the object.

[0171] The terminal determines the component with the highest similarity as the search result by comparing the object's features with the object features of each object stored in the database.

[0172] In step 1304, the terminal generates text based on the search results.

[0173] The terminal generates prompts based on the search results.

[0174] In step 1305, the terminal inputs the search object and prompt words into the large language model.

[0175] The terminal inputs the search target and prompt words into the large language model, which then outputs the component's operating environment information and explanatory information.

[0176] After the search target and prompt words are input into the large language model at the terminal, the large language model can output something like this: Figure 14 The results shown Figure 14 This is a schematic diagram of the search results provided in an embodiment of this application.

[0177] Search results can be as follows Figure 14 The expected result of the user command shown does not require any action; only detection is needed. Some tips can help you complete the user command. Tips are as follows: If you want to click an application icon, use the action "Open Application." If you want to exit the application, use the action "Home." If a pop-up is detected on the current page, please close it first. You should first look for controls from the component information provided below. The following control information is accurate:

[0178] Name: " / Friend / FriendList / Tab / CommonBtn"; Text: This switches to the leaderboard list control.

[0179] Name: " / Center / LobbyBottom / Newbie / Interactor"; Text: This is a smart assistant control; Related keywords: Miss Ji.

[0180] Name: " / Form_Lobby_SysTray / Center / PlayerBtn / OBBtn"; Text: This is the spectator button control.

[0181] The following is combined Figure 15 Explain the subsequent procedures. Figure 15 This is a schematic diagram of the data processing flow provided in the embodiments of this application.

[0182] In step 1501, the terminal inputs data into the intelligent agent.

[0183] The terminal inputs the collected game page (i.e., the virtual scene mentioned above) environmental data (i.e., the first operation information mentioned above) and supplementary data (i.e., the first description information mentioned above) into the intelligent agent.

[0184] In step 1502, the agent performs operation steps based on the received information.

[0185] The intelligent agent combines the user's input operation instructions (i.e. the instructions to be executed mentioned above) and breaks them down into individual operation steps (for example, if the instruction to be executed is to send an email, it is broken down into the operation steps of entering the email system, selecting the recipient, determining the email content, and sending the email).

[0186] The agent selects and calls the corresponding tools (such as click tools, swipe tools, etc.) to execute the corresponding operation steps by thinking about the operations required for the corresponding steps. After the operation steps are completed, they are saved as historical operation steps to provide reference data for the next decision.

[0187] In step 1503, the multiple agents jointly execute the instructions.

[0188] Because the automated operation process in games is quite complex, the responsibilities of the agents are broken down. This multi-agent collaborative approach improves stability. The agents are primarily divided into decision-making agents, progress-summarizing agents, assertion agents, and reflection agents. The collaboration process among these agents is as follows: Figure 16 As shown, Figure 16 This is a schematic diagram of the collaborative process of the intelligent agents provided in the embodiments of this application.

[0189] In step 1601, the decision-making agent makes a decision.

[0190] After receiving the operating environment information and explanatory information, the decision-making agent makes a corresponding decision, decomposes the instruction to be executed into corresponding operation steps, and executes the corresponding operation steps.

[0191] In step 1602, the reflective agent is used to perform consistency checks.

[0192] After the decision-making agent completes each operation step, the reflective agent performs a consistency check on the execution results of the operation steps.

[0193] In step 1603, the progress summary agent determines the next operation steps.

[0194] After the reflective agent confirms that the consistency check of the current operation step has passed, the execution progress summary agent determines the next operation step, and the decision agent executes the next operation step.

[0195] In step 1604, an assertion is made to determine whether the intelligent agent has completed the execution of the control instruction.

[0196] After all the operation steps are completed, the assertion agent determines whether the final execution result is the same as expected.

[0197] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0198] The following description continues to illustrate the exemplary structure of the virtual scene data processing device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the data processing device 555 storing the virtual scene in the memory 540 may include:

[0199] The acquisition module 5551 is used to acquire the first operating environment information of the virtual scene and query the first description information of the first operating environment information;

[0200] The decomposition module 5552 is used to decompose the instruction to be executed to obtain at least one operation step corresponding to the instruction to be executed.

[0201] Execution module 5553 is used to execute the operation steps based on the first operating environment information and the first description information of the first operating environment information;

[0202] The detection module 5554 is used to perform consistency detection on the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step in response to the completion of the at least one operation step, and obtain a consistency detection result.

[0203] In some embodiments, the acquisition module 5551 is further configured to acquire the component type and component coordinates of the component in the virtual scene; perform position verification processing on the component coordinates of the component in the virtual scene to obtain the verified component coordinates; and combine the component type of the component and the verified component coordinates of the component to form the first operating environment information.

[0204] In some embodiments, the acquisition module 5551 is further configured to determine a first coordinate and a second coordinate in the component coordinates that indicate the two ends of the diagonal of the component, wherein the diagonal endpoint corresponding to the first coordinate is located to the upper left of the diagonal endpoint corresponding to the second coordinate; when any of the following conditions are met, it is determined that the component corresponding to the component coordinate exceeds the range of the virtual scene, and the component is removed from the virtual scene: the abscissa of the first coordinate is greater than the abscissa of the second coordinate; the ordinate of the first coordinate is greater than the ordinate of the second coordinate; the abscissa of the first coordinate is less than the minimum abscissa of the scene coordinates; the ordinate of the first coordinate is less than the minimum ordinate of the scene coordinates; the abscissa of the second coordinate is greater than the maximum abscissa of the scene coordinates; the ordinate of the second coordinate is greater than the maximum ordinate of the scene coordinates; and the remaining components in the virtual scene after removal are subjected to position verification processing to obtain the verified component coordinates.

[0205] In some embodiments, the acquisition module 5551 is further configured to perform the following processing for each remaining component: when the component is located in a sliding control in the virtual scene, and the horizontal coordinate of the first coordinate is less than the horizontal coordinate of the third coordinate, the horizontal coordinate of the first coordinate is updated based on the horizontal coordinate of the third coordinate to obtain a verified first coordinate; when the component is located in a sliding control in the virtual scene, and the vertical coordinate of the first coordinate is less than the vertical coordinate of the third coordinate, the vertical coordinate of the first coordinate is updated based on the vertical coordinate of the third coordinate to obtain a verified first coordinate; when the component is located in a sliding control in the virtual scene, and the horizontal coordinate of the second coordinate is greater than the horizontal coordinate of the fourth coordinate, the horizontal coordinate of the second coordinate is updated based on the horizontal coordinate of the fourth coordinate to obtain a verified second coordinate; when the component is located in a sliding control in the virtual scene, and the vertical coordinate of the second coordinate is greater than the vertical coordinate of the fourth coordinate, the vertical coordinate of the second coordinate is updated based on the vertical coordinate of the fourth coordinate to obtain a verified second coordinate.

[0206] In some embodiments, the acquisition module 5551 is further configured to perform page pop-up window recognition processing on the virtual scene to obtain a pop-up window recognition result; when the pop-up window recognition result indicates that there is a page pop-up window in the virtual scene, the first operating environment information of the page pop-up window in the virtual scene is acquired.

[0207] In some embodiments, the acquisition module 5551 is further configured to acquire the component coordinates of the components in the first operating environment information; perform feature extraction processing on the component coordinates to obtain the position features of the components; and retrieve the first description information of the first operating environment information from the database based on the position features of the components.

[0208] In some embodiments, the execution module 5553 is further configured to determine the target component involved in the operation steps within the virtual scene; determine the component information of the target component and the first description information of the target component from the first operation environment information of the virtual scene and the first description information of the first operation environment information; and, based on the component information of the target component and the first description information of the target component, invoke an operation tool to perform the operation steps on the target component.

[0209] In some embodiments, the detection module 5554 is further configured to obtain the expected image corresponding to the expected execution result in the virtual scene and the actual image corresponding to the actual execution result in the virtual scene; extract a first image feature of the expected image and a second image feature of the actual image; and use the feature similarity between the first image feature and the second image feature as the consistency detection result of the consistency detection.

[0210] In some embodiments, the execution module 5553 is further configured to: when the execution result of the operation step is correct, obtain the execution progress of the instruction to be executed, and continue to execute subsequent operation steps of the operation step based on the execution progress; when the execution result of the operation step is incorrect, obtain the second operation environment information of the virtual scene, and query the second description information of the second operation environment information; wherein, the second operation environment information is the real-time operation environment information of the virtual scene after the operation step is executed; determine the correction operation step corresponding to the operation step based on the execution result; and execute the correction operation step corresponding to the operation step based on the second operation environment information and the second description information of the second operation environment information.

[0211] In some embodiments, the detection module 5554 is further configured to: obtain third operating environment information of the virtual scene and query third explanatory information of the third operating environment information when the consistency detection result indicates that the expected execution result is inconsistent with the actual execution result; wherein the third operating environment information is the real-time operating environment information of the virtual scene after executing the at least one operation step; determine the correction operation steps corresponding to the instruction to be executed based on the difference between the actual execution result and the expected execution result; and execute the correction operation steps corresponding to the instruction to be executed based on the third operating environment information and the third explanatory information of the third operating environment information.

[0212] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the virtual scene data processing method described above in this application.

[0213] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the data processing method for a virtual scene provided in this application. For example, ... Figure 3A The data processing method for the virtual scene is shown.

[0214] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0215] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0216] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0217] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0218] In summary, the embodiments of this application can achieve the following beneficial effects:

[0219] The system acquires the first operating environment information of the virtual scene and queries the first description information of the first operating environment information. It then decomposes the instruction to be executed into at least one operation step corresponding to the instruction. By decomposing the instruction, it breaks it down into at least one operation step, which is then executed sequentially to complete the instruction. If an error occurs in a certain operation step, the impact of the error can usually be limited to that step and will not affect the entire instruction. Furthermore, when a problem occurs, it can more quickly and accurately locate which specific step has an error, reducing the cost of error correction. Based on the first operating environment information and the first description information of the first operating environment information, the operation steps are executed. In response to the completion of the at least one operation step, a consistency check is performed on the expected execution result of the instruction and the actual execution result of the at least one operation step to obtain a consistency check result. By checking the consistency between the actual execution result and the expected execution result, it is ensured that the final result meets the user's needs, guaranteeing the accuracy of the execution operation. Simultaneously, by automatically executing the steps decomposed from the instruction using the operating environment information in the virtual scene, it avoids the need for manual execution by the user, improving the convenience of performing operations in the virtual scene.

[0220] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method for a virtual scene, characterized in that, The method includes: Obtain the first operating environment information of the virtual scene, and query the first description information of the first operating environment information; The instruction to be executed is decomposed to obtain at least one operation step corresponding to the instruction to be executed; Based on the first operating environment information and the first descriptive information of the first operating environment information, the operation steps are executed; In response to the completion of the at least one operation step, a consistency check is performed on the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step to obtain a consistency check result.

2. The method according to claim 1, characterized in that, The acquisition of the first operating environment information of the virtual scene includes: Obtain the component type and component coordinates of the components within the virtual scene; The component coordinates within the virtual scene are subjected to position verification processing to obtain the verified component coordinates; The component type of the component and the verified component coordinates of the component are combined to form the first operating environment information.

3. The method according to claim 2, characterized in that, The step of performing position verification processing on the component coordinates within the virtual scene to obtain the verified component coordinates includes: Determine a first coordinate and a second coordinate in the component coordinate system that indicate the two ends of the diagonal of the component, wherein the diagonal endpoint corresponding to the first coordinate is located to the upper left of the diagonal endpoint corresponding to the second coordinate; If any of the following conditions are met, the component corresponding to the component coordinates is determined to be outside the range of the virtual scene, and the component is removed from the virtual scene: The x-coordinate of the first coordinate is greater than the x-coordinate of the second coordinate; The ordinate of the first coordinate is greater than the ordinate of the second coordinate; The x-coordinate of the first coordinate is less than the minimum x-coordinate of the scene coordinates; The ordinate of the first coordinate is less than the minimum ordinate of the scene coordinates; The x-coordinate of the second coordinate is greater than the maximum x-coordinate of the scene coordinates; The ordinate of the second coordinate is greater than the maximum ordinate of the scene coordinates; The remaining components in the virtual scene after removal are subjected to position verification processing to obtain the coordinates of the verified components.

4. The method according to claim 3, characterized in that, The sliding control in the virtual scene includes a third coordinate and a fourth coordinate indicating the two ends of the diagonal of the sliding control, and the diagonal endpoint corresponding to the third coordinate is located to the upper left of the diagonal endpoint corresponding to the fourth coordinate. The step of performing position verification processing on the remaining components in the virtual scene after removal to obtain the verified component coordinates includes: Perform the following processing for each remaining component: When the component is located in the sliding control in the virtual scene, and the horizontal coordinate of the first coordinate is less than the horizontal coordinate of the third coordinate, the horizontal coordinate of the first coordinate is updated based on the horizontal coordinate of the third coordinate to obtain the verified first coordinate. When the component is located in a sliding control in the virtual scene, and the ordinate of the first coordinate is less than the ordinate of the third coordinate, the ordinate of the first coordinate is updated based on the ordinate of the third coordinate to obtain the verified first coordinate. When the component is located in a sliding control in the virtual scene, and the horizontal coordinate of the second coordinate is greater than the horizontal coordinate of the fourth coordinate, the horizontal coordinate of the second coordinate is updated based on the horizontal coordinate of the fourth coordinate to obtain the verified second coordinate. When the component is located in a sliding control in the virtual scene, and the ordinate of the second coordinate is greater than the ordinate of the fourth coordinate, the ordinate of the second coordinate is updated based on the ordinate of the fourth coordinate to obtain the verified second coordinate.

5. The method according to claim 1, characterized in that, The acquisition of the first operating environment information of the virtual scene includes: Perform pop-up recognition processing on the virtual scene to obtain pop-up recognition results; When the pop-up recognition result indicates that a page pop-up exists in the virtual scene, the first operating environment information of the page pop-up in the virtual scene is obtained.

6. The method according to claim 1, characterized in that, The first descriptive information for querying the first operating environment information includes: Obtain the component coordinates of the components in the first operating environment information; The component coordinates are subjected to feature extraction processing to obtain the position features of the component; Based on the location characteristics of the component, the database is searched to obtain the first description information of the first operating environment information.

7. The method according to claim 1, characterized in that, The first operating environment information includes component information of components within the virtual scene; the step of performing the operation steps based on the first operating environment information and the first descriptive information of the first operating environment information includes: Identify the target components involved in the operation steps within the virtual scene; From the first operating environment information of the virtual scene and the first descriptive information of the first operating environment information, determine the component information of the target component and the first descriptive information of the component information of the target component; Based on the component information of the target component and the first descriptive information of the component information of the target component, the operation tool is invoked to perform the operation steps on the target component.

8. The method according to claim 1, characterized in that, The consistency check between the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step includes: Obtain the expected screen corresponding to the expected execution result in the virtual scene and the actual screen corresponding to the actual execution result in the virtual scene; Extract the first image features of the expected image and the second image features of the actual image; The feature similarity between the first image feature and the second image feature is used as the consistency detection result.

9. The method according to claim 1, characterized in that, For any of the aforementioned operation steps, after performing the operation step, the method further includes: When the execution result of the operation step is correct, the execution progress of the instruction to be executed is obtained, and the subsequent operation steps of the operation step are continued to be executed based on the execution progress; When the execution result of the operation steps is incorrect, obtain the second operation environment information of the virtual scene, and query the second description information of the second operation environment information; The second operating environment information is the real-time operating environment information of the virtual scene after the operation steps are executed; Based on the execution result, determine the corrected operation steps corresponding to the operation steps; Based on the second operating environment information and the second explanatory information of the second operating environment information, the correction operation steps corresponding to the operation steps are executed.

10. The method according to claim 1, characterized in that, After performing a consistency check on the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step, the method further includes: When the consistency detection result indicates that the expected execution result is inconsistent with the actual execution result, the third operating environment information of the virtual scene is obtained, and the third description information of the third operating environment information is queried. The third operating environment information is the real-time operating environment information of the virtual scene after the execution of the at least one operating step; Based on the difference between the actual execution result and the expected execution result, determine the correction operation steps corresponding to the instruction to be executed; Based on the third operating environment information and the third explanatory information of the third operating environment information, the correction operation steps corresponding to the instruction to be executed are performed.

11. A data processing device for a virtual scene, characterized in that, The device includes: The acquisition module is used to acquire the first operating environment information of the virtual scene and query the first description information of the first operating environment information; The decomposition module is used to decompose the instruction to be executed to obtain at least one operation step corresponding to the instruction to be executed. The execution module is configured to execute the operation steps based on the first operating environment information and the first descriptive information of the first operating environment information; The detection module is used to perform consistency detection on the expected execution result of the instruction to be executed and the actual execution result of the at least one operation step in response to the completion of the operation step, and to obtain a consistency detection result.

12. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions or computer programs stored in the memory, implements the data processing method for the virtual scene according to any one of claims 1 to 10.

13. A computer-readable storage medium storing executable instructions or a computer program, characterized in that, When the executable instructions are executed by the processor, they implement the data processing method for the virtual scene according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the data processing method for the virtual scene according to any one of claims 1 to 10.

Images