Data processing method, device, equipment and computer program

Abstract

This application discloses a data processing method, an apparatus, a device, and a readable storage medium, the method including the steps of: acquiring first image data transmitted from a first client and storing it in a receiving queue; performing image recognition processing on the first image data in the receiving queue; storing second image data continuously acquired in the image recognition processing process of the first image data transmitted from the first client in the receiving queue to acquire an updated receiving queue; and, when a first object area where an object is located in the first image data is extracted by the image recognition processing, transmitting the first object image data included in the first object area to a target cloud application server, and synchronously performing image recognition processing on the second image data having the latest reception timestamp in the updated receiving queue. This application can reduce the delay in image transmission and improve the efficiency of image recognition.

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on September 24, 2021, bearing application number 202111123508.7 and entitled "Data Processing Method, Apparatus, Device and Readable Storage Medium", the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and readable storage medium. [Background technology]

[0003] With the rapid development of electronic device-related technologies and the widespread use of the Internet, the game industry, which exists and operates on electronic devices, has grown by leaps and bounds. In particular, the emergence of smart devices such as smartphones and tablets has revealed the game industry's great potential for growth.

[0004] In order to provide the user with an immersive experience, a corresponding virtual object (e.g., a virtual video object) is constructed in the cloud game based on the user's human body image, and the virtual object is displayed in the cloud game, that is, the virtual object places the user in a virtual cloud game scenario, and the user has an immersive experience in the cloud game. Generally, in this process, the terminal captures the user's screen through a camera, and then directly recognizes and extracts the user's human image on the terminal, thereby obtaining and displaying the corresponding virtual object. Summary of the Invention [Problem to be solved by the invention]

[0005] Because the computing power of the terminal is not high, it leads to the problem of reduced efficiency of image recognition due to insufficient computing power. Furthermore, there is a large delay when the terminal transmits the person image recognition result to the cloud, and there is also a delay when the game displays the virtual object, which may cause the virtual action of the virtual object displayed in the game to not match the user's current action state.

[0006] The embodiments of the present application provide a data processing method, an apparatus, a device, and a readable storage medium, which can reduce the delay in image transmission and improve the efficiency of image recognition. [Means for solving the problem]

[0007] One aspect of an embodiment of the present application provides a data processing method executed by a computing device, the method comprising: Acquiring first image data sent from a first client and storing it in a receiving queue, the first image data being image data including the acquired object when the first client executes a cloud application; performing image recognition processing on the first image data in the receiving queue, and storing second image data continuously acquired during the image recognition processing of the first image data and transmitted from the first client in the receiving queue to acquire an updated receiving queue; The method includes a step of: when a first object area in which the object is located in the first image data is extracted by the image recognition process, sending the first object image data contained in the first object area to a target cloud application server and synchronously performing image recognition processing on the second image data having a last received timestamp in the update receive queue, wherein the target cloud application server renders the first object image data to obtain rendering data and sends the rendering data to the first client.

[0008] One aspect of an embodiment of the present application provides a data processing device arranged in a computing device, the device comprising: A data acquisition module that acquires first image data transmitted from a first client and stores the first image data in a receiving queue, where the first image data is image data including an acquired object when the first client executes a cloud application; an image recognition module that performs image recognition processing on the first image data in the receiving queue; a queue update module for storing second image data continuously acquired and transmitted from the first client in a receiving queue during an image recognition process of the first image data, and acquiring an updated receiving queue; an area transmission module that transmits the first object image data included in the first object area to a target cloud application server when a first object area in which the object is located in the first image data is extracted by the image recognition process, the target cloud application server renders the first object image data to obtain rendering data, and transmits the rendering data to the first client; The area sending module further synchronously performs image recognition processing on the second image data having the last received timestamp in the update receiving queue.

[0009] One aspect of an embodiment of the present application provides another data processing method executed by a computer device, the method comprising: A step of receiving first object image data transmitted from a traffic server and storing the first object image data in a first buffer zone in the buffer zone set whose operating state is in a storage state, wherein the first object image data is image data included in a first object area, the first object area being an area in which an acquired object is located in the first image data after the traffic server performs image recognition processing on the first image data, and the first image data is transmitted from a first client to the traffic server, and when the first client runs a cloud application, the first image data is image data including the acquired object; when a second buffer zone in the set of buffer zones, the operational state of which is in a read state, does not contain unprocessed object image data, adjusting the operational state of the first buffer zone to a read state, adjusting the operational state of the second buffer zone to a store state, reading a first object area from the first buffer zone, the operational state of which is in a read state, and performing a rendering process on the first object area; receiving second object image data transmitted from a traffic server during rendering of the first object area and storing the second object image data in a second buffer zone whose operating state is in a memory state, the second object image data being image data included in the second object area, the second object area being obtained by the traffic server extracting the first object area and then performing image recognition processing on the second image data, the second object area being an area in which the object is located in the second image data, the second image data being image data having a last reception timestamp obtained from an update receiving queue when the traffic server extracts the first object area, the second image data in the update receiving queue being continuously obtained from the first client during the traffic server performing image recognition processing on the first image data; When rendering data corresponding to the first object image data is acquired, the method includes a step of adjusting the operation state of the first buffer zone to a storage state, adjusting the operation state of the second buffer zone to a reading state, reading the second object image data from the second buffer zone whose operation state is in the reading state, and performing a rendering process on the second object image data.

[0010] One aspect of an embodiment of the present application provides another data processing device arranged in a computer device, the device comprising: an area storage module that receives first object image data transmitted from a traffic server and stores it in a first buffer zone in the buffer zone set whose operating state is in a storage state, the first object image data being image data included in a first object area, the first object area being an area in which an acquired object is located in the first image data after the traffic server performs image recognition processing on the first image data, the first image data being transmitted from a first client to the traffic server, and the area storage module being image data including the acquired object when the first client executes a cloud application; an area rendering module for, when a second buffer zone in the set of buffer zones, whose operational state is in a read state, does not contain unprocessed object image data, adjusting an operational state of the first buffer zone to a read state, adjusting an operational state of the second buffer zone to a store state, reading a first object area from the first buffer zone, whose operational state is in the read state, and performing a rendering process on the first object area; an area receiving module that receives second object image data transmitted from a traffic server during a rendering process of a first object area and stores the second object image data in a second buffer zone whose operating state is in a storage state, the second object image data being image data included in the second object area, the second object area being obtained by the traffic server extracting the first object area and then performing image recognition processing on the second image data, the second object area being an area in which an object is located in the second image data, the second image data being image data having a last reception timestamp obtained from an update receiving queue when the traffic server extracts the first object area, the second image data in the update receiving queue being continuously obtained from a first client during the process in which the traffic server performs image recognition processing on the first image data; and a state adjustment module that, when rendering data corresponding to the first object image data is acquired, adjusts the operation state of the first buffer zone to a storage state, adjusts the operation state of the second buffer zone to a reading state, reads the second object image data from the second buffer zone whose operation state is in the reading state, and performs a rendering process on the second object image data.

[0011] One aspect of an embodiment of the present application provides a computer device, comprising a processor and a memory, The memory stores a computer program which, when executed by the processor, causes the processor to perform the methods in the embodiments of the present application.

[0012] One aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored therein, the computer program including program instructions that, when executed by a processor, perform a method in an embodiment of the present application.

[0013] According to one aspect of the present application, there is provided a computer program product or computer program, the computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computing device reading the computer instructions from the computer readable storage medium, and the processor executing the computer instructions, thereby causing the computing device to perform a method provided by any one of the aspects of the embodiments of the present application. Effect of the Invention

[0014] In the embodiment of the present application, when a client (e.g., a first client) obtains first image data including an object, the first image data is sent to a related computer device (e.g., a traffic server), and the traffic server performs image recognition processing, so that image recognition does not need to be performed locally on the client, and the traffic server with high computing power performs image recognition processing on the first image data, thereby improving image recognition efficiency and clarity; in addition, in the present application, the traffic server stores the received first image data in a receiving queue, and in the process of performing image recognition processing on the first image data, continuously synchronously obtains second image data from the first client, stores the second image data in a receiving queue, and obtains an updated receiving queue. That is, when the traffic server in the present application performs image recognition processing on the first image data, it does not suspend the reception of the second image data, and the receiving queue can realize the synchronous progress of image processing and image reception, thereby reducing image transmission delay. Furthermore, when the traffic server extracts a first object region where an object is located in the first image data through image recognition processing, the traffic server transmits the first object image data included in the first object region to the target cloud application server, and the target cloud application server performs rendering and transmits the rendering data to the first client for display in the cloud application. After extracting the first object region, the traffic server can obtain the second image data having the last received timestamp in the receiving queue, and continue to process the second image data.As can be seen from this, in the present application, after performing image recognition processing on certain image data, image data having a last received timestamp is obtained from the receiving queue and processed, and image data is not recognized one by one according to the time order of the receiving timestamps, so that the recognition efficiency of the image data can be improved, and since the image data having the last received timestamp is collected based on the current action of the object, even when image recognition and display are performed on the image data having the last received timestamp, it matches with the current action of the object. As described above, the present application can improve the image recognition efficiency, reduce the delay in image transmission, and ensure that the virtual action of a virtual object displayed in a cloud application matches the current action state of the object. [Brief description of the drawings]

[0015] In order to clearly explain the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the necessary drawings in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings on the premise that they do not make efforts worthy of inventive step. [Figure 1] FIG. 1 is a network architecture diagram provided by an embodiment of the present application. [Figure 2a] FIG. 1 is a schematic diagram of a scenario provided by an embodiment of the present application. [Figure 2b] FIG. 1 is a schematic diagram of a scenario provided by an embodiment of the present application. [Diagram 3] 1 is a schematic flow diagram of a data processing method provided by an embodiment of the present application; [Figure 4] FIG. 2 is a schematic diagram of a frame skip process provided by an embodiment of the present application; [Diagram 5] 1 is a schematic flow diagram of a data processing method provided by an embodiment of the present application; [Figure 6] FIG. 1 is a schematic diagram of a site fusion scenario provided by an embodiment of the present application. [Figure 7]FIG. 11 is a flow diagram of sending first object image data to a target cloud application server provided by an embodiment of the present application; [Figure 8] 1 is a schematic flow diagram of a data processing method provided by an embodiment of the present application; [Figure 9] FIG. 2 is a schematic diagram of a double buffer zone state change provided by an embodiment of the present application; [Figure 10] FIG. 1 is a system architecture diagram provided by an embodiment of the present application. [Figure 11] FIG. 1 is a schematic diagram of a system flow provided by an embodiment of the present application. [Figure 12] 1 is an interaction flowchart provided by an embodiment of the present application; [Figure 13] FIG. 1 is a structural schematic diagram of a data processing device provided by an embodiment of the present application; [Figure 14] FIG. 2 is a structural schematic diagram of another data processing device provided by an embodiment of the present application; [Figure 15] FIG. 2 is a structural schematic diagram of a computer device provided by an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, the technical solutions of the embodiments of the present application will be clearly and completely described in conjunction with the drawings of the embodiments of the present application, of course, the described embodiments are not all the embodiments, but only some of the embodiments of the present application. Based on the embodiments of the present application, on the premise that a person skilled in the art does not make an effort worthy of inventive step, all other embodiments obtained fall within the protection scope of the present application.

[0017] Please refer to FIG. 1, which is a network architecture diagram provided by an embodiment of the present application. As shown in FIG. 1, the network architecture may include a traffic server 1000, a terminal device cluster and a cloud application server cluster 10000, where the terminal device cluster may include one or more terminal devices, where the number of terminal devices is not limited. As shown in FIG. 1, the multiple terminal devices may include terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n, where the terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n respectively establish network connections with the traffic server 1000, and each terminal device performs data interaction with the traffic server 1000 through the network connection. The cloud application server cluster 10000 may include one or more cloud application servers, where the number of cloud application servers is not limited. 1, the multiple cloud application servers may include cloud application server 10001, cloud application server 10002, ..., cloud application server 1000n, and as shown in FIG. 1, cloud application server 10001, cloud application server 10002, ..., cloud application server 1000n each have a network connection with traffic server 1000, and each cloud application server performs data interaction with traffic server 1000 through the network connection. Each cloud application server may be a cloud application server, and one terminal device corresponds to one cloud application server (multiple terminal devices may correspond to the same cloud application server), and when the terminal device runs a cloud application, the corresponding cloud application server provides corresponding functional services (e.g., computing services).For example, if the cloud application is a cloud game application, the cloud application server is a cloud game server, and when a terminal device runs the cloud game application, the corresponding cloud game server provides corresponding functional services.

[0018] Here, a cloud application may be installed on each terminal device in FIG. 1, and when the cloud application is executed by each terminal device, it performs data interaction with the traffic server 1000 in FIG. 1, and the traffic server 1000 can receive traffic data from each terminal device.

[0019] The cloud applications include applications having a function of displaying data information such as text, image, audio, and video. For example, the cloud applications may be entertainment applications (e.g., game applications), and the entertainment applications may be used for user game entertainment. The traffic server 1000 in the present application obtains traffic data through these cloud applications, and for example, the traffic data may be image data (which may be called first image data) including a user (which may be called an object) captured by a camera component of a terminal device.

[0020] Then, after acquiring the first image data, the traffic server 1000 stores the first image data in a receiving queue, acquires the first image data from the receiving queue, and performs image recognition processing on the first image data. Here, after acquiring and transmitting the first image data to the traffic server 1000, the terminal device continuously acquires image data (which may be called second image data) including an object, and in the process of performing image recognition processing on the first image data, the traffic server 1000 further continuously acquires the second image data acquired by the terminal device. Similar to the first image data, the traffic server 1000 may store the second image data in a receiving queue to acquire an updated receiving queue including one or more pieces of second image data. Here, if the first image data is the first image data received by the traffic server 1000, the traffic server 1000 does not need to store the first image data in the receiving queue, but directly performs image recognition processing on the first image data, and during the image recognition processing, continuously acquires the second image data captured by the terminal device (i.e., the second image data, third image data, fourth image data, etc. after the first image data), and stores the second image data in the receiving queue.

[0021] The traffic server 1000 extracts an area (which may be called a first object area) in which an object is located in the first image data by image recognition processing, and then obtains image data (which may be called first object image data) included in the first object area from the first image data, and transmits the first object image data to a cloud application server corresponding to the terminal device, and the cloud application server reads and renders the first object image data, and after the rendering is completed and the rendering data is obtained, transmits the rendering data to the terminal device, and the terminal device displays and outputs the rendering data in the cloud application. In addition, when the traffic server 1000 extracts the first object area in which an object is located in the first image data, it continues to perform image recognition processing on other image data, for example, the traffic server 1000 obtains second image data (i.e., the last received image data) having a last received timestamp (also called a "latest received timestamp") from an update receiving queue including the second image data, and performs image recognition processing on the second image data (which may be called target image data) having the last received timestamp. Here, when the traffic server 1000 performs image recognition processing on the target image data, it continuously acquires image data (which may be called third image data) including an object from the terminal device, stores the third image data in an updated receiving queue, and acquires a new receiving queue. When the traffic server 1000 extracts an area (which may be called a second object area) in which an object is located in the target image data, it may acquire image data (which may be called second object image data) included in the second object area from the target image data, and further transmits the second object image data to a cloud application server corresponding to the terminal device.The traffic server 1000 also obtains third image data (which may be called new target image data) having a last received timestamp from the current new receiving queue, and performs image recognition processing on the new target image data.

[0022] Here, the traffic server 1000 in the present application can continuously receive other image data while performing image recognition processing on certain image data, and thus does not need to wait for reception after recognition is completed, and can realize synchronization between recognition and reception, thereby reducing image data reception delay. In addition, every time recognition of certain image data is completed (after extracting the area where the object is located), the traffic server performs frame skip processing (i.e., instead of obtaining the next image data (image data with the closest reception timestamp) of the image data currently processed and performing image recognition processing, obtaining the image data with the current last reception timestamp and performing image recognition processing), thereby reducing image data queuing delay by frame skip processing. If the image data with the last reception timestamp is the collected user's current behavior, image recognition processing is performed on the image data with the last reception timestamp and displayed, and the rendering data displayed in the cloud application is synchronized and matched with the user's current behavior.

[0023] In an embodiment of the present application, one terminal device is selected from multiple terminal devices to perform data interaction with the traffic server 1000, and the terminal device may include, but is not limited to, a smart terminal having a media data processing function (e.g., a video data playback function, a music data playback function), such as a smartphone, a tablet, a laptop, a desktop PC, a smart TV, a smart speaker, a desktop computer, a smart watch, a smart in-vehicle device, etc. For example, in an embodiment of the present application, the above-mentioned cloud application may be integrated into the terminal device 100a in FIG. 1, and in this case, the terminal device 100a performs data interaction with the traffic server 1000 through the cloud application.

[0024] Here, the method provided by the embodiments of the present application may be executed by a computer device, and the computer device may include, but is not limited to, a user terminal or a traffic server. The traffic server may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud application server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, big data and artificial intelligence platforms.

[0025] The terminal device and the traffic server may be directly or indirectly connected by wire or wireless communication, which is not limited in the present application.

[0026] Here, the above computer devices (for example, the above traffic server 1000, the terminal device 100a, the terminal device 100b, etc.) may be one node in a distributed system, and the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the nodes in the form of network communication. A peer-to-peer (P2P) network can be formed between the nodes, and the P2P protocol is an application layer protocol that runs on the Transmission Control Protocol (TCP) protocol. In the distributed system, any type of computer device, such as a traffic server, a terminal device, or other electronic device, can join the peer-to-peer network to become one node in the blockchain system. For ease of understanding, the concept of blockchain will be explained below, that is, blockchain is a new application mode of computer technology such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm, and mainly organizes data according to time order and encrypts it into a ledger, making it impossible to tamper with and counterfeit, and at the same time, data can be verified, stored, and updated. When the computer device is a blockchain node, the data in the present application (such as the first image data, the first object area, the second object image data, etc.) are provided with authenticity and security due to the immutability and anti-counterfeiting characteristics of the blockchain, which makes the results obtained by performing data processing based on these data more reliable.

[0027] Please refer to Fig. 2a, which is a schematic diagram of a scenario provided by an embodiment of the present application. The terminal device 100a in Fig. 2a may be the terminal device 100a in the terminal device cluster 100 in the embodiment corresponding to the above Fig. 1, the traffic server 1000 in Fig. 2a may be the traffic server 1000 in the embodiment corresponding to the above Fig. 1, and the cloud application server 10001 in Fig. 2a may be the cloud application server 10001 in the embodiment corresponding to the above Fig. 1.

[0028] As shown in FIG. 2a, the terminal device 100a includes a game application, and after a user a (which may be called object a) launches the game application, the terminal device 100a collects a screen including object a (which may be called original image frame 20a) by the camera component 200a, and performs encoding processing (e.g., H264 encoding processing) on ​​the original image frame to obtain image data. The terminal device 100a transmits the image data to the traffic server 1000. The traffic server 1000 stores the image data in a receiving queue, then retrieves the image data from the receiving queue, and performs decoding processing on the image data to obtain the original image frame 20a. Here, the traffic server 1000 stores each image data (including the first image data) received in the receiving queue, but may directly perform decoding processing on the first image data received by the traffic server 1000 without storing it. For example, if the image data corresponding to the above original image frame 20a is the first image data received by the traffic server 1000, the traffic server 1000 may store it in the receiving queue according to the storage rules and then retrieve it from the receiving queue, or it may not store it and directly perform a decoding process to retrieve the original image frame 20a.

[0029] The traffic server 1000 performs image recognition processing on the original image frame 20a, and determines an object edge curve P1 corresponding to the object a in the original image frame 20a through the image recognition processing. Here, the user a continuously generates a motion action (e.g., raising both hands, shaking head, squatting, etc.), so that the terminal device 100a continuously collects original image frames including the object a through the camera component 200a after collecting the original image frame 20a, and each time an original image frame including the object a is successfully obtained, the terminal device 100a performs an encoding process thereon to obtain image data and transmits it to the traffic server 1000. Currently, even if the traffic server 1000 performs image recognition processing on the original image frame 20a, it does not affect the reception of image data, and in the process of image recognition processing, the traffic server 1000 continuously obtains different image data from the terminal device 100a and temporarily stores these image data in a receiving queue.

[0030] After determining the object edge curve P1, the traffic server 1000 extracts all areas covered by the object edge curve P1 (which may be called object areas P2) from the original image frame 20a, and obtains all image contents included in the object areas from the original image frame 20a, thereby obtaining all image contents included in the object areas (which may be called object image data). The traffic server 1000 obtains a cloud application server (e.g., cloud application server 10001) corresponding to the terminal device 100a, and transmits the object image data included in the object areas P2 to the cloud application server 10001. After obtaining the object image data, the cloud application server 10001 performs a rendering process on the object image data to obtain rendering data P3, and the cloud game server transmits the rendering data P3 to the corresponding terminal device 100a.

[0031] Please refer to Fig. 2b, which is a schematic diagram of a scenario provided by an embodiment of the present application. As shown in Fig. 2b, after receiving the rendering data P3, the terminal device 100a displays the rendering data P3 in a game application. As shown in Fig. 2b, the corresponding virtual environment of the game (which may be a game scenario) includes a virtual background (virtual house background), a dancing virtual object 2000a (dancing), and a dancing virtual object 2000b, and the rendering data P3 may be displayed in the virtual environment.

[0032] Here, when the object region P2 is extracted from the original image frame 20a, the traffic server 1000 further processes the image data in the receiving queue. In order to match the rendering data displayed on the terminal device 100a with the current action of the object a, the traffic server 1000 performs frame skip processing, i.e., the traffic server 1000 obtains the image data with the last received timestamp in the receiving queue and performs decoding and image recognition processing. The image data with the last received timestamp is the last image data sent by the terminal device 100a at the current time, and the image data corresponds to the latest real-time action of the object a. Then, after extracting, rendering and outputting the corresponding object region, the displayed rendering data matches the actual action of the object.

[0033] Here, if the cloud application is a game application, the object a is a game player, and the corresponding character image rendering data (e.g., rendering data P3) is displayed in the game application, that is, the player character image is projected into the game scenario, so that the game player can be "located" in the game scenario and the immersive feeling of the game player can be improved. In addition, in the embodiment of the present application, the reception queue realizes synchronization between image recognition and reception, so that the reception delay of image data can be reduced, and the frame skip processing can speed up the recognition efficiency of image data, further reducing the delay and improving the matching rate between the player character image displayed in the game and the player.

[0034] Referring to Fig. 3, Fig. 3 is a flow diagram of a data processing method provided by an embodiment of the present application. The method is executed by a computer device, which may be executed by a terminal device (e.g., any one of the terminal devices in the terminal device cluster in Fig. 1 above, such as the terminal device 100a) or a traffic server (e.g., the traffic server 1000 in Fig. 1 above), or may be executed jointly by the terminal device and the traffic server by including a terminal device and a traffic server. In this embodiment, the method is described as being executed by the traffic server as an example. The data processing method includes at least the following S101 to S103.

[0035] S101: Obtain first image data sent from a first client and store it in a receiving queue, where the first image data is image data including an obtained object when the first client runs a cloud application.

[0036] In this application, the first client may be a terminal device, and an application program is arranged on the first client, and the application program may be a cloud application (e.g., a game application), etc. Taking a cloud application as an example, when using the first client, a user launches the cloud application on the first client, for example, the user clicks the cloud application and clicks the launch control to run the cloud application. The first client may be any one of the clients.

[0037] Here, after the user launches the cloud application in the first client, that is, when the first client runs the cloud application, the first client collects a screen including the user (which may be called an object) through the camera component, and the screen including the user may be called an original image frame. The first client performs an encoding process on the original image frame to obtain an image encoding file, and the image encoding file may be called image data. The first client transmits the image data to a traffic server (the traffic server is a server having an image decoding function and an image recognition function, and obtains the encoded file transmitted from the first client and performs decoding and image recognition processing). Among different encoding methods, as an encoding format, the H264 encoding method has a high compression rate, and after performing H264 encoding on the same image, it occupies less bandwidth during the transmission process, so that H264 is widely applied in mobile video applications. Thus, in this application, in order to reduce the transmission bandwidth between the first client and the traffic server, the H264 encoding method is preferably selected for the encoding method of the original image frame. Of course, the encoding method used by the first client to encode the original image frame may be any method capable of encoding the original image frame other than H264, such as H262 encoding method, H263 encoding method, H265 encoding method, etc., and the present application is not limited thereto.

[0038] In this application, after obtaining image data transmitted from the first client, the traffic server stores the image data in a receiving queue. Taking the first image data as an example, after obtaining the first image data, the traffic server stores the first image data in a receiving queue, and the specific method is as follows. Receive the first image data transmitted from the first client (the first image data is data obtained by the first client performing an encoding process on the original image frame), and then the traffic server obtains a receiving timestamp at which the first image data is received, and associates the first image data with the receiving timestamp and stores it in the receiving queue. That is, when storing each image data, the traffic server also stores its receiving time. For example, for image data A, if the receiving time (which may be the receiving timestamp) of the traffic server is 19:09 on September 5, 2021, the traffic server associates the image data A with the receiving time, which is 19:09 on September 5, 2021, and stores it in the receiving queue. Here, if no image data is stored in the receiving queue, the receiving queue may be empty.

[0039] S102: Perform image recognition processing on the first image data in the receiving queue, and store second image data continuously acquired during the image recognition processing of the first image data and transmitted from the first client in the receiving queue to acquire an updated receiving queue.

[0040] In this application, after storing first image data in a receiving queue, the first image data is obtained from the receiving queue and image recognition processing is performed. Since the first image data is actually an image encoding file, the first image data is decoded to restore it and obtain an original image frame, and then image recognition processing is performed on the original image frame. The specific method is as follows: the first image data is decoded to obtain decoded image data having an initial image format, format conversion is performed on the decoded image data to obtain an original image frame having a standard image format, and image recognition processing is performed on the original image frame having the standard image format.

[0041] Here, the standard image format is a predetermined image format for performing image recognition processing uniformly. For example, if the predetermined image for performing image recognition processing has a color format (Red Green Blue color mode, RGB color mode), the RGB format is called the standard image format. After performing a decoding process on the first image data, the decoded image data having the initial image format is obtained, and if the initial image format is the standard image format, the decoded image data is determined to be an original image frame having the standard image format. If the initial image format is different from the standard image format, the initial image format is converted into the standard image format to obtain an original image frame having the standard image format. For example, if the initial image format is a YUV format, the YUV format is converted into an RGB format to obtain an original image frame having the RGB format.

[0042] The traffic server performs decoding to obtain an original image frame corresponding to the first image data, and then performs image recognition processing on the original image frame having the standard image format to determine the area in which the object is located in the original image frame (which may be referred to as the first object area).For a specific implementation form of performing image recognition processing to determine the first object area, please refer to the description of the embodiment corresponding to Figure 5 below.

[0043] Here, in this application, after sending the first image data to the traffic server, the first client continuously collects a screen (a new original image frame) including an object, and performs an encoding process on each original image frame to obtain an image encoding file (which may be called second image data). The first client continuously transmits each second image data to the traffic server. In the process of performing an image recognition process on the first image data, the traffic server does not suspend the reception of image data, but continuously obtains the second image data from the first client, and temporarily stores the second image data in a receiving queue to obtain an updated receiving queue.

[0044] S103: If a first object area where the object is located in the first image data is extracted through the image recognition process, the first object area is sent to a target cloud application server, and the image recognition process is synchronized with the second image data having the last received timestamp in the update receiving queue, and the target cloud application server renders the first object area to obtain rendering data, and sends the rendering data to the first client.

[0045] In the present application, the first object region where the object is located in the original image frame is determined by the above image recognition process, and then the image content (which may be called first object image data) contained in the first object region is obtained from the original image frame, and the traffic server extracts the first object region and the first object image data contained in the first object region. When the first object region and the first object image data are extracted, the traffic server obtains a target cloud application server corresponding to the first client and sends the first object image data to the target cloud application server. Note that the target cloud application server may be a cloud application server corresponding to the first client, and when the first client executes a cloud application, the cloud application server provides the first client with computing services, such as central processing unit (CPU) computing services, graphics processing unit (GPU) computing services, etc. The target cloud application server performs a rendering process on the first object image data to obtain rendering data corresponding to the first object image data, and transmits the rendering data to the first client, and the first client displays the rendering data in a cloud application.

[0046] Here, after extracting the first object region, the traffic server may further perform decoding and image recognition processing on other image data. For example, the traffic server obtains the second image data having the last reception timestamp in the update reception queue, and performs decoding and image recognition processing on the second image data.

[0047] The process in which the traffic server obtains the second image data having the last reception timestamp from the update reception queue and performs the decoding and image recognition process is also called frame skip processing. Also refer to FIG. 4, which is a schematic diagram of the frame skip processing provided by the embodiment of the present application. As shown in FIG. 4, the reception queue 40a includes image data 1, image data 2, image data 3, image data 4, image data 5, image data 6, image data 7, image data 8, and image data 9, and the image data 1 to image data 9 are sorted in the order of the earliest reception timestamp, and are sequentially indicated by symbols 1, 2, 3, 4, 5, 6, 7, 8, and 9 in FIG. 4, that is, the reception timestamp of the image data 1 is the first reception timestamp, and the reception timestamp of the image data 9 is the last timestamp. The image data 1, image data 2, image data 3, image data 4, and image data 5 are image data after processing, the image data 6 is image data currently being processed by the traffic server, the image data 7, image data 8, and image data 9 are image data received when the traffic server processes the image data 6, and the image data 7, image data 8, and image data 9 are queued and wait for processing.

[0048] As shown in Fig. 4, when the traffic server extracts an object area where an object is located in the image data 6, the traffic server obtains image data 9 from the tail of the receiving queue 40a (i.e., obtains image data having the last received time stamp), skips image data 7 and image data 8, and performs decoding and image recognition processing on the image data 9, which is a frame skip processing process. For ease of understanding, an example will be given below to describe in detail. It is assumed that the time required for the traffic server to decode and perform image recognition processing is 30 ms, and the time interval for the traffic server to receive two sets of image data is 10 ms (i.e., 0 ms, receive one image data, 10 ms, receive the next image data, and 20 ms, receive the next image data). In the process of processing the image data 6, the traffic server continuously receives image data 7 (in this case, stored at the tail of the receiving queue after image data 6), image data 8 (in this case, stored at the tail of the receiving queue after image data 7), and image data 9 (in this case, stored at the tail of the receiving queue after image data 8). Then, after processing image data 6, the receiving queue can be referred to receive queue 40a, in which case the traffic server directly gets the most recent image data (i.e., image data 9) from the end of the receiving queue and skips image data 7 and image data 8 (image data 7 and image data 8 have not been processed, but are skipped without actually being processed, so image data 7 and image data 8 are determined to be processed image data).

[0049] Similarly, as shown in Fig. 4, when processing image data 9, the traffic server continuously receives image data 10 and image data 11 (indicated by symbols 10 and 11 in Fig. 4) to obtain a receiving queue 40b, and then, when an object area in the image data 9 of an object is extracted, the traffic server obtains the image data at the end of the receiving queue 40b (i.e., the image data 11 having the last received timestamp), and the traffic server skips the image data 10 and performs decoding and image recognition processing on the image data 11. Similarly, when processing image data 11, the traffic server continuously receives other image data to obtain a receiving queue 40c, and even after processing the image data 11, it obtains the image data at the end of the receiving queue 40c, and thus repeats the process, and the process is not repeated here.

[0050] Here, each time an image data item is processed, the image data that is queued up in front of it (i.e., processed image data) is cleared, thereby expanding the storage space of the receiving queue. For example, if the processed image data in the receiving queue 40a includes image data 1 to 5, image data 1 to 5 are deleted, and in this case, the receiving queue 40a includes only image data 6 to 9. In other words, each time an image data item is processed, image data having a last reception timestamp is obtained from the receiving queue, and then the image data (which may be called history image data) that is queued up before the image data (i.e., whose reception timestamp is earlier than that of the image data) is deleted. For example, when image data 9 to be processed is obtained from the receiving queue 40a, the history image data (including image data 1 to 8) before the image data 9 has already been deleted.

[0051] That is, when the first object area in the first image data is extracted, the processed image data (including the first image data) is cleared, and the second image data having the last received timestamp is obtained from the remaining unprocessed image data. Similarly, when the first object area in the first image data is extracted, the second image data having the last received timestamp is obtained, and then the history image data having an earlier received timestamp than the second image data is deleted and cleared (that is, after the second image data having the last received timestamp in the update receiving queue is obtained, the image recognition process is performed on the second image data, and the history image data in the update receiving queue is synchronously deleted, and the history image data is the image data in the update receiving queue having an earlier received timestamp than the second image data). Here, any queue clearing method is for improving the storage space of the queue, and the present application does not specifically limit the queue clearing method.

[0052] Here, after the traffic server receives the encoded code stream (i.e., image data) transmitted from the first client, if the period of decoding and image recognition processing (hereinafter referred to as image processing) is greater than the reception interval period of both frame image data (for example, the period of image processing is 30 ms and the reception interval period of both frame image data is 10 ms), when there is no receiving queue, the first client will wait for the traffic server to perform image processing on the current image data, which will significantly increase the delay in image data transmission and greatly affect the transmission efficiency of image data. In this application, the receiving queue stores image data, and in the image processing process, the traffic server continuously receives image data transmitted from the first client, but when the traffic server performs image processing on the image data in order, the image data recognized by the traffic server does not match the latest state of the object very well, and the image data recognized by the traffic server is significantly delayed, so that the frame skip processing is used to perform image processing on the latest image data every time the traffic server performs image processing, thereby reducing the delay in image recognition and improving the image recognition efficiency due to the high computing ability of the traffic server.

[0053] In an embodiment of the present application, when a client (e.g., a first client) obtains first image data including an object, the client transmits the first image data to a related computer device (e.g., a traffic server), and the traffic server performs image recognition processing, so that image recognition does not need to be performed at the locale of the first client, and the traffic server having high computing power performs image recognition processing on the first image data, thereby improving image recognition efficiency and clarity. In addition, in the present application, the traffic server stores the received first image data in a receiving queue, and in the process of performing image recognition processing on the first image data, continuously synchronously obtains second image data from the first client, stores the second image data in a receiving queue, and obtains an updated receiving queue. That is, when the traffic server in the present application performs image recognition processing on the first image data, it does not suspend the reception of the second image data, and the receiving queue can realize the synchronous progress of image processing and image reception, thereby reducing image transmission delay. Furthermore, when the traffic server extracts a first object region where an object is located in the first image data through image recognition processing, the traffic server can send the first object image data included in the first object region to the target cloud application server, and the target cloud application server can perform rendering and send the rendering data to the first client to display in the cloud application. After extracting the first object region, the traffic server can obtain the second image data having the last received timestamp in the receiving queue, and continue to process the second image data.As can be seen from this, in the present application, after performing image recognition processing on certain image data, image data having a last received timestamp is obtained from the receiving queue and processed, instead of recognizing image data one by one according to the time order of the receiving timestamps, the recognition efficiency of the image data can be improved, and since the image data having the last received timestamp is collected based on the current action of the object, even when image recognition and display are performed on the image data having the last received timestamp, it matches with the current action of the object. As described above, the present application can improve the image recognition efficiency, reduce the delay in image transmission, and ensure that the virtual action of a virtual object displayed in a cloud application matches the current action state of the object.

[0054] Please refer to Fig. 5, which is a flow diagram of a data processing method provided by an embodiment of the present application. This flow corresponds to the embodiment corresponding to Fig. 3 above, and Fig. 5 can be referred to for the flow of performing image recognition processing on an original image frame to determine a first object region, and this flow includes at least the following S501-S503.

[0055] S501: Recognize object edge keypoints in an original image frame of an object.

[0056] Here, the object edge keypoint is the object contour keypoint of the object, and if the original image frame contains a key part of the object, the object contour keypoint can be the contour keypoint of the key part. If the key part is the head, the object edge keypoint is the head contour keypoint, and if the key part is the neck, the object edge keypoint is the neck contour keypoint.

[0057] Recognition of object edge keypoints of an object may be performed by artificial intelligence algorithms, a dedicated Graphics Processing Unit (GPU), etc., to which the present application is not limited.

[0058] S502: Connect the object edge key points to obtain an object edge curve of the object.

[0059] After the object edge key points are determined, the object edge points are connected (e.g., every two adjacent points are connected) to obtain an object edge curve (object contour) corresponding to the object. Exemplarily, referring to the above scenario example of FIG. 2a, the curve P1 in FIG. 2a is the object contour of object a.

[0060] S503: Determine an area covered by the object edge curve in the original image frame as an initial object area in the original image frame of the object, and determine a first object area based on the initial object area.

[0061] After determining the object edge curve of object a, the area covered by the object edge curve can be determined from the original image frame, and the area is determined as the first object area where the object is located in the original image frame. For example, refer to the above scenario example of Fig. 2a, as shown in Fig. 2a, in the original image frame 20a, the area covered by the object edge curve P2 is area P2 (the area P2 is the area where object a is located), and the area P2 is determined as the area where the object is located in the original image frame (here, may be referred to as the first object area).

[0062] In a possible embodiment, the area covered by the above object edge curve is also called the initial object area. After determining the initial object area, the initial object area is not determined as the final first object area for a while, and the first object area is determined based on the initial object area. The specific method is as follows: first obtain the object key portion of the object displayed in the initial object area, then obtain the object recognition placement information for the object and the object recognition placement portion indicated by the object recognition placement information, and match the object recognition placement portion with the object key portion. If the object recognition placement portion matches with the object key portion, execute the step of determining the first object area based on the initial object area. If the object recognition placement portion does not match with the object key portion, it is determined that the first object area cannot be extracted by the image recognition processing.

[0063] Here, after determining the initial object region of the object, obtain the object key part of the object displayed in the initial object region (the object key part may be a body part of the object, for example, head, neck, arm, abdomen, leg, foot, etc.), and then obtain the object recognition placement part of the object that needs to be included in the original image frame (i.e., a recognition rule, which specifies the part of the object that needs to be included in the original image frame collected by the terminal device). Take the object recognition placement part as an example that the leg is the object recognition placement part, if it is specified that the user's leg must be included in the original image frame collected by the terminal device, after performing decoding and image recognition processing on the received image data, the object key part displayed in the extracted initial object region is the head and neck, so that the object key part (head and neck) does not match the object recognition placement part (leg), and it is determined that the key part of the object displayed in the original image frame does not meet the requirement, in this case, it can be directly determined that the first object region cannot be extracted by the image recognition processing (i.e., the part does not meet the requirement, and extraction fails). If it is specified that the original image frame collected by the terminal device must include the user's legs, then after performing decoding and image recognition processing on the received image data, if the object key part displayed in the extracted initial object area is the legs, it can be determined that the key part of the object displayed in the original image frame meets the requirement, and in this case, the initial object area is determined to be the first object area.

[0064] In the implementation of a possible embodiment, according to the above introduction, after determining that the object recognition location portion does not match the object key portion and the first object region cannot be extracted by image recognition processing, the traffic server obtains the image data next to the current image data in the update receiving queue, and performs image processing on the next image data. For example, if the current image data is the first image data, the traffic server obtains the image data next to the first image data in the receiving queue (i.e., the image data having the first receiving timestamp in the image data whose receiving timestamp is later than that of the first image data), and then performs image processing on the next image data. The specific method is as follows: the second image data having the first receiving timestamp in the update receiving queue is determined as the image data to be recognized, and the image recognition processing is performed on the image data to be recognized, and when the image recognition processing extracts the object region to be processed in which the object is located in the image data to be recognized, the object region to be processed is sent to the target cloud application server.

[0065] In fact, if the traffic server determines that the first object area cannot be extracted by image recognition processing when the image processing period for the current image data (e.g., the first image data) is short enough and the efficiency is fast enough, the traffic server obtains the next image data (image data having the first reception timestamp) of the current image data and performs image processing on the next image data. The purpose is as follows: when the user performs an action, the first client obtains an image frame, encodes it, and then transmits it to the traffic server. If the traffic server quickly recognizes that the object key part contained therein does not conform to the standard and the object area and the image data contained in the object area cannot be extracted, the cloud application server cannot receive the extracted object image data and cannot render and display it. In this case, the traffic server performs image processing on the next image data to extract the object area of ​​the next image data, and then transmits the object image data contained therein to the cloud application server, where it renders and outputs it. This reduces the jumping of the user's human image displayed on the cloud application and improves its continuity. Of course, after it is determined that the first object area cannot be extracted by the image recognition process, the second image data having the last received timestamp (not the image data having the first received timestamp) may be directly obtained from the current update receiving queue and image processing may be performed. Regardless of the obtaining method, any of them is a possible processing method after it is determined that the first object area cannot be extracted by the image recognition process, and the specific processing method may be manually set according to the actual situation, and the present application does not limit it.

[0066] In the implementation of the possible embodiment, according to the above introduction, when it is determined that the object recognition arrangement portion matches the object key portion, the initial object region may be directly determined as the first object region. In addition, when it is determined that the object recognition arrangement portion matches the object key portion, the specific method of determining the first object region may be as follows: obtain the object key portion of the object displayed in the initial object region, and if the object key portion has portion completeness, determine the initial object region as the first object region; if the object key portion does not have portion completeness, obtain N (N is a positive integer) sample image frames in the sample database, obtain a sample image frame to be processed corresponding to the object from the N sample image frames, and determine the first object region according to the sample image frame to be processed and the initial object region.

[0067] The specific method for determining the first object area based on the sample image frame to be processed and the initial object area is as follows: after obtaining overall part information in the sample image frame to be processed, a part area to be merged is determined from the overall part information based on the object key part, and the part area to be merged and the initial object area are merged to obtain the first object area.

[0068] Here, the present application collects the complete human image sample data of the entire user (complete human image sample data from head to foot) in advance, one user corresponds to one sample image frame, and one sample image frame displays the complete human image data of one user. In this way, when it is determined that the object recognition arrangement part matches the object key part after extracting the initial object area, it can determine whether the object key part has part completeness, and if it has part completeness, it directly determines the initial object area as the first object area; if the object key part does not have completeness, it obtains the sample image frame to be processed corresponding to the object in the sample database, and then obtains the whole part information of the object from the sample image frame to be processed, and fills the initial object area to be complete according to the whole part information to obtain a complete first object area including the complete part.

[0069] For ease of understanding, please also refer to FIG. 6, which is a schematic diagram of a part fusion scenario provided by an embodiment of the present application. As shown in FIG. 6, the initial object area is the initial object area 600a, and the object key parts displayed in the initial object area 600a include head, neck, arm, chest, and abdomen (i.e., the upper body parts of the user), and the object recognition and placement parts are also the upper body parts of the user, that is, the first client needs to collect the upper body parts of the user, and the initial object area meets the requirement. Furthermore, it is determined whether the object key part has part completeness, and in this case, if the part completeness is defined as the user's overall human image completeness (i.e., including the upper body parts and the lower body parts, i.e., from the head to the feet), the object key part displayed in the initial object area 600a does not have part completeness. In this case, the traffic server 1000 obtains a sample image frame to be processed (e.g., the sample image frame to be processed 600b) corresponding to the object in the sample database. 6, the whole part information displayed in the sample image frame to be processed includes complete information from the head to the feet of the object, and in this case, since the initial object region 600a already includes the upper body part, the lower body part in the sample image frame to be processed is determined as the part region to be merged (i.e., region 600c), and the part region 600c to be merged is extracted. Furthermore, the part region 600c to be merged and the initial object region 600a are merged (for example, stitched) to obtain a first object region 600d including the upper body part and the lower body part.Here, by collecting the user's entire body part information (e.g., from head to feet) in advance, the first client does not need to strictly require the user to stand in a fixed position to collect the entire body part every time it captures the user's screen, and the user can move flexibly, and the first client only needs to acquire partial body part information. After acquiring partial body part information of the first client, the traffic server can supplement and stitch based on the entire body part information collected in advance to acquire the complete body part, thereby improving the user's experience and immersion.

[0070] Here, in the above process, if the part integrity is defined as the integrity of the user's upper body part, the object key part displayed in the initial object area 600a actually already has part integrity, and thus the initial object area 600a is directly determined to be the first object area.

[0071] The specific method of acquiring the sample image frame to be processed corresponding to the object in the above sample database is as follows, and may use a face matching method. When collecting the sample image frames of the user, the corresponding sample image frames are matched by a user identifier (e.g., user name, user number, etc.), so that each sample image frame has a user identifier (also called a sample identifier). When sending image data to the traffic server, the first client carries and transmits the user identifier contained in the image data, and the traffic server matches the corresponding sample image frame to be processed according to the carried user identifier and the sample identifier of the sample image frame. The specific implementation form of acquiring the sample image frame to be processed corresponding to the object in the sample database is not limited to the above-mentioned form, and the present application does not limit the specific implementation form.

[0072] In an embodiment of the present application, when a client (e.g., a first client) obtains first image data including an object, the client transmits the first image data to a related computer device (e.g., a traffic server), and the traffic server performs image recognition processing, so that image recognition does not need to be performed locally on the client, and the traffic server having high computing power performs image recognition processing on the first image data, thereby improving image recognition efficiency and clarity. In addition, in the present application, the traffic server stores the received first image data in a receiving queue, and in the process of performing image recognition processing on the first image data, continuously synchronously obtains second image data from the first client, stores the second image data in a receiving queue, and obtains an updated receiving queue. That is, when the traffic server in the present application performs image recognition processing on the first image data, it does not suspend the reception of the second image data, and the receiving queue can realize the synchronous progress of image processing and image reception, thereby reducing image transmission delay. Furthermore, when the traffic server extracts a first object region where an object is located in the first image data through image recognition processing, the traffic server can send the first object image data included in the first object region to the target cloud application server, and the target cloud application server can perform rendering and send the rendering data to the first client to display in the cloud application. After extracting the first object region, the traffic server can obtain the second image data having the last received timestamp in the receiving queue, and continue to process the second image data.As can be seen from this, in the present application, after performing image recognition processing on certain image data, image data having a last received timestamp is obtained from the receiving queue and processed, and image data is not recognized one by one according to the time order of the receiving timestamps, so that the recognition efficiency of the image data can be improved, and since the image data having the last received timestamp is collected based on the current action of the object, even when image recognition and display are performed on the image data having the last received timestamp, it matches with the current action of the object. As described above, the present application can improve the image recognition efficiency, reduce the delay in image transmission, and ensure that the virtual action of a virtual object displayed in a cloud application matches the current action state of the object.

[0073] In a possible embodiment, when each client runs a cloud application for the first time, the cloud application server corresponding to each client sends a registration request to the traffic server, and the registration request requests the traffic server to register the device. After registration, the traffic server adds the device identifier corresponding to the cloud application server to the stored device identifier set to prove that the cloud application server is a registered cloud application server. If it is a registered cloud application server, it indicates that it is a legitimate cloud application server, and in this case, the traffic server can perform data interaction with the legitimate cloud application server. Then, when the first client sends image data (e.g., the first image data) to the traffic server, it sends the device identifier (also called the device identifier to be confirmed) of the cloud application server (also called the binding cloud application server) having a binding relationship, so that the traffic server checks whether it is registered (whether it is legitimate) by the device identifier, and if it determines that the binding cloud application server is registered, it determines the binding cloud application server as a target cloud application server and sends the first object image data to the target cloud application server. In other words, after determining the first object area as described above, before sending the first object image data to the target cloud application server, the traffic server first determines whether the cloud application server corresponding to the first client is registered, and if it is determined that the cloud application server is registered, sends the first object image data to the corresponding target cloud application server.

[0074] For ease of understanding, please refer to Fig. 7, which is a flow diagram of sending first object image data to a target cloud application server provided by an embodiment of the present application. The flow is described by taking as an example that the first image data carries a device identifier to be confirmed (the device identifier to be confirmed is the device identifier of the binding cloud application server, and the binding cloud application server and the first client have a binding relationship). As shown in Fig. 7, the flow at least includes the following S701 to S704.

[0075] S701: Obtain a stored device identifier set, where the stored device identifier set includes M stored device identifiers, where one stored device identifier corresponds to one registered cloud application server, and M is a positive integer.

[0076] When each client runs a cloud application (generally when running a cloud application for the first time), the cloud application server corresponding to each client sends a registration request to the traffic server, which requests the traffic server to register the device; after registration, the traffic server adds the device identifier corresponding to the cloud application server to a set of stored device identifiers, thereby proving that the cloud application server is a registered cloud application server. Taking the client as an example, the specific method is as follows: when a user uses a client to launch a cloud application, the second client generates an application launch notification in response to the application launch operation, and sends the application launch notification to a corresponding cloud application server (also referred to as a cloud application server to be registered); in this case, the cloud application server to be registered sends a registration request to a traffic server based on the application launch notification; the traffic server receives the registration request sent from the cloud application server to be registered, and detects device index information of the cloud application server to be registered based on the registration request; if the device index information meets the processing quality condition, obtains a device identifier to be stored of the cloud application server to be registered and stores it in a stored device identifier set, converts the cloud application server to be registered into a registered cloud application server, and converts the device identifier to be stored into a stored device identifier.

[0077] In addition, the device index information includes network quality parameters, device version, functional module quality indicators, storage space indicators, etc., and the detection of the device index information may be detecting whether a certain indicator is passed, for example, detecting whether a network quality parameter is passed, and if the network quality parameter is passed, the device index information of the cloud application server to be registered is deemed to satisfy the processing quality conditions. The detection of the device index information may also be detecting whether two or more indicators are all passed, and it is confirmed that the device index information of the cloud application server to be registered satisfies the processing quality conditions only under the condition that all indicators are passed.

[0078] The following takes the example of device index information including network quality parameters and device version as an example to describe a specific method for detecting device index information of a cloud application server to be registered. The specific method is as follows: based on a registration request, the network quality parameters and device version of the cloud application server to be registered are obtained; when the network quality parameters reach a parameter threshold and the device version matches the quality standard version (quality passing version), it is determined that the device index information meets the processing quality conditions; when the network quality parameters do not reach the parameter threshold or the device version does not match the quality standard version, it is determined that the device index information does not meet the processing quality conditions.

[0079] As can be seen from the above, the stored device set stores stored device identifiers corresponding to different registered cloud application servers, and after obtaining the device identifier to be confirmed sent from the first client, the stored device identifier set can be obtained and the device identifier to be confirmed can be matched with the stored device identifier set.

[0080] S702: Determine whether there is a stored device identifier that matches the device identifier to be confirmed among the M stored device identifiers.

[0081] If the answer is YES, the following step S703 is executed, and if the answer is NO, the following step S704 is executed.

[0082] S703: If there is a stored device identifier that matches the device identifier to be confirmed among the M stored device identifiers, it is determined that the binding cloud application server indicated by the device identifier to be confirmed belongs to the registered cloud application server, the binding cloud application server indicated by the device identifier to be confirmed is determined as a target cloud application server, and the first object image data is sent to the target cloud application server.

[0083] The device identifier to be confirmed is matched with M stored device identifiers, and if there is a stored device identifier among the M stored device identifiers that matches the device identifier to be confirmed, it is determined that the binding cloud application server indicated by the device identifier to be confirmed belongs to the registered cloud application server, and in this case, the binding cloud application server is determined to be the target cloud application server, and the first object image data is sent to the target cloud application server.

[0084] S704: When the stored device identifier does not exist in the M stored device identifiers, generate device abnormality prompt information and send it to the first client.

[0085] The device identifier to be checked is matched with M stored device identifiers, and when there is no stored device identifier matching the device identifier to be checked in the stored device identifier set, it is determined that the binding cloud application server indicated by the device identifier to be checked belongs to an unregistered cloud application server, and the binding cloud application server is not registered, and the traffic server cannot send the first object image data to the binding cloud application server. In this case, the traffic server generates device abnormality prompt information (server unregistered prompt information), and the traffic server returns the device abnormality prompt information to the first client, and the first client sends a registration notification to the corresponding binding cloud application server based on the device abnormality prompt information, and the binding cloud application server requests registration from the traffic server based on the registration notification.

[0086] Here, in reality, the client that transmits image data to the traffic server is not limited to the above first client, and may have different clients, and different clients correspond to different cloud application servers, so that there are multiple cloud application servers to register with the traffic server (there are multiple cloud application services connected to the traffic server). In this way, in the present application, the stored device identifier set is stored in advance, and when the client transmits image data, it carries and transmits the device identifier of the corresponding cloud game server, so that the correspondence between the client and the cloud application server can be determined, and whether the cloud application server is registered can be determined, so that the user screen collected by the client can be transmitted to the correct registered cloud application server, and the accuracy of the user screen displayed in the cloud application can be improved.

[0087] Here, for each cloud application server, for example a target cloud application server, when receiving person image data (i.e., object image data, for example, first object image data) transmitted from a traffic server, the typical processing steps include the following three: 1. A receive buffer zone is allocated, and the person image data (the above-mentioned first object image data) is written into the receive buffer zone. 2. After the reception is completed, the person image data is copied and processed and rendered. 3. After you finish rendering, go back to step 1.

[0088] That is, the target cloud application server first writes the character image data into the buffer zone, reads and renders the character image data, and after the rendering is completed, continues to receive the character image data and writes it into the buffer zone. Since the amount of data received for high-resolution character image data is large, when the target cloud application server allocates the buffer zone and copies the data, it consumes a large amount of time, which greatly affects the subsequent reception time of the character image data and causes a large amount of delay. Thus, although the image reception delay on the traffic server side can be reduced by the frame skip processing of the traffic server described above, the cloud application server side still has a delay problem. In order to further reduce the delay, the present application provides a data processing method, which allocates a double buffer zone on the cloud application server side. For ease of understanding, please also refer to FIG. 8, which is a flow diagram of a data processing method provided by an embodiment of the present application, the flow is executed by a computing device, and the computing device may be a target cloud application server, for example, a cloud game server, and the flow corresponds to a data processing process after the target cloud application server receives the object image data. As shown in FIG. 8, the flow at least includes the following S801 to S804.

[0089] S801: Receive first object image data transmitted from a traffic server and store it in a first buffer zone in the buffer zone set whose operating state is in a memory state, the first object image data being image data included in a first object area, the first object area being an area in the first image data in which an object acquired after the traffic server performs image recognition processing on the first image data is located, and the first image data being transmitted from a first client to the traffic server and including an object acquired when the first client executes a cloud application.

[0090] For the process of determining and extracting the first object area and the first object image data, please refer to the description of S102 in the embodiment corresponding to Fig. 3 above, and will not be repeated here. After extracting the first object area and obtaining the first object image data, the traffic server sends the first object image data to the target cloud application server, and the target cloud application server stores the first object image data in the first buffer zone in the buffer zone set whose operating state is in the storage state.

[0091] In addition, the target cloud application server pre-allocates two receiving buffer zones (Buffer) of the same size, and sets the operating state of one of the buffer zones to a storage state, that is, the buffer zone is actually a storage buffer zone, and the target cloud application server stores the received data in the storage buffer zone; at the same time, the operating state of the other buffer zone is set to a read state, that is, the buffer zone is actually a read buffer zone, and when data needs to be read and rendered, the target cloud application server reads from the read buffer zone.

[0092] Taking two buffer zones as an example, a specific method of allocating double buffer zones and generating a buffer zone set is as follows: first, a first buffer zone and a second buffer zone are allocated, and an initial pointer identifier of the first buffer zone is set to a store pointer identifier, and an initial pointer identifier of the second buffer zone is set to a read pointer identifier, where the operation state of the first buffer zone having the store pointer identifier is a store state, and the operation state of the second buffer zone having the read pointer identifier is a read state, and then a buffer zone set is generated based on the first buffer zone whose operation state is a store state and the second buffer zone whose operation state is a read state. In this case, when the first object image data is received, the operation state of the first buffer zone is in a store state, so the first object image data is stored in the first buffer zone.

[0093] S802: If a second buffer zone in the buffer zone set whose operational state is a read state does not contain unprocessed object image data, the operational state of the first buffer zone is adjusted to a read state, the operational state of the second buffer zone is adjusted to a store state, the first object image data is read from the first buffer zone whose operational state is a read state, and a rendering process is performed on the first object image data.

[0094] According to the above introduction, if the initial working state of the first buffer zone is a storage state and the initial working state of the second buffer zone is a read state, when the first image data is received and stored, the target cloud application server synchronously reads and renders the data stored in the second buffer zone (also called stored image data). At this time, when no image data is stored in the second buffer zone, the second buffer zone does not contain unprocessed object image data, and after storing the first object image data in the first buffer zone, the storage pointer identifier of the first buffer zone is switched to a read pointer identifier, and the storage pointer identifier of the second buffer zone is switched to a storage pointer identifier, so that the working states of the first buffer zone and the second buffer zone are exchanged, and the current working state of the first buffer zone is a read state, and the current working state of the second buffer zone is a storage state. In this case, the first object image data is read from the first buffer zone and a rendering process is performed, and the second object image data is subsequently received and stored in the second buffer zone.

[0095] Here, by setting up a double buffer zone, the target cloud application server does not need to wait for rendering to be completed before receiving the remaining data, and can realize synchronization between reading and receiving, thereby greatly reducing the receiving delay.

[0096] Here, according to the above introduction, if the initial operating state of the first buffer zone is a storage state and the initial operating state of the second buffer zone is a read state, when the first image data is received and stored, the target cloud application server synchronously reads and renders the data stored in the second buffer zone (also called stored image data). When image data is stored in the second buffer zone, but the reading and rendering of all image data is completed, the processed image data in the second buffer zone is cleared, and in this case, it is determined that the second buffer zone does not contain unprocessed image data, and the pointer identifier is switched to adjust the operating state of the first buffer zone to a read state and adjust the operating state of the second buffer zone to a storage state. Then, the first object image data is read from the first buffer zone in the read state and rendering processing is performed.

[0097] S803: In the process of rendering the first object image data, second object image data sent from the traffic server is received and stored in a second buffer zone whose operating state is in a memory state, the second object image data is image data included in a second object area, the second object area is obtained by the traffic server extracting the first object area and then performing image recognition processing on the second image data, the second object area is an area in which the object is located in the second image data, the second image data is image data having a last received timestamp obtained from the update receiving queue when the traffic server extracts the first object area, and the second image data in the update receiving queue is continuously obtained from the first client in the process of the traffic server performing image recognition processing on the first image data.

[0098] The second object area may be an area extracted after the traffic server performs image recognition processing on the second image data, and the second object image data may be image data included in the second object area in the second image data. The specific extraction method may be the same as the extraction method of the first object area, and will not be repeated here.

[0099] During the process of rendering the first object image data, the second object image data sent from the traffic server is received, and during the process of actually reading the data, the target cloud application server synchronously receives the data and stores it in the second buffer zone currently in the storage state, thereby reducing delay.

[0100] S804: When rendering data corresponding to the first object image data is acquired, the operation state of the first buffer zone is adjusted to a storage state, the operation state of the second buffer zone is adjusted to a reading state, the second object image data is read from the second buffer zone whose operation state is in the reading state, and a rendering process is performed on the second object image data.

[0101] When reading and rendering of the data stored in the first buffer zone is completed (for example, when the first buffer zone contains only the first object image data, even if rendering data corresponding to the first object image data is obtained), the operation state of the first buffer zone is adjusted to a storage state, and the operation state of the second buffer zone is adjusted to a reading state, in which case the second object image data may be read from the second buffer zone in the reading state and a rendering process may be performed on the second object image data, or other object image data may be synchronously received and stored in the first buffer zone in the storage state. Note that a specific implementation form of adjusting the operation state of the first buffer zone to a storage state and adjusting the operation state of the second buffer zone to a reading state may be a pointer identifier switching form. The specific method is as follows: when first rendering data corresponding to a first object area is obtained, a read pointer identifier corresponding to the first buffer zone, which indicates a read state, and a memory pointer identifier corresponding to the second buffer zone, which indicates a memory state, are obtained, the read pointer identifier corresponding to the first buffer zone is switched to a memory pointer identifier, the operating state of the first buffer zone having the memory pointer identifier is a memory state, and the memory pointer identifier of the second buffer zone is switched to a read pointer identifier, and the operating state of the second buffer zone having the read pointer identifier is a read state.

[0102] For ease of understanding, please also refer to FIG. 9, which is a schematic diagram of a double buffer zone state change provided by an embodiment of the present application. As shown in FIG. 9, take the first buffer zone as buffer zone 900a and the second buffer zone as buffer zone 900b as an example. In this case, the operation state of buffer zone 900a is a read state, and the object image data stored in buffer zone 900a includes object image data 1 to object image data 10, which are indicated by symbols 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 in FIG. 9. The object image data 1 to object image data 7 are data that have been read, and object image data 8 to object image data 10 are data to be read. In addition, the operating state of the buffer zone 900b is a storage state, and in the process of reading data from the buffer zone 900a, the target cloud application server continuously receives object image data and stores it in the buffer zone 900b. In this case, the data received in the buffer zone 900b includes object image data 11 to object image data 14 (there are six remaining spatial positions in the buffer zone 900b for receiving object image data).

[0103] As shown in FIG. 9, when the reading of data in the buffer zone 900a is completed (i.e., the reading of object image data 7 to object image data 9 is completed), the buffer zone 900a is cleared, and in this case, the data received in the buffer zone 900b includes object image data 11 to object image data 20 (indicated by symbols 11, 12, 13, 14, 15, 16, 17, 18, 19, and 20 in FIG. 9). Furthermore, the operation state of the buffer zone 900a is switched to a storage state, and the operation state of the buffer zone 900b is switched to a read state, so that the target cloud application server reads data from the buffer zone 900b (for example, reads data in order from object image data 11), and at the same time, the target cloud application server synchronously receives the object image data and stores it in the buffer zone 900a. For example, new object image data 1 to new object image data 3 are received and then stored in the buffer zone 900a.

[0104] Here, the above buffer zone 900a and buffer zone 900b are merely illustrative examples for ease of understanding, and do not have any actual reference meaning.

[0105] In the embodiment of the present application, the frame skip process of the traffic server can reduce the image reception delay on the traffic server side and improve the image recognition efficiency, and the double buffer zone allocation process of the target cloud application server can eliminate the need to copy data, and can simply switch the working state of the two buffer zones (e.g., pointer switching), eliminating the need to allocate buffer zones every time, and data reception and data processing (e.g., reading and rendering data) can be performed simultaneously, without the need to wait for each other, thereby reducing the delay. In other words, in reducing the delay on the traffic server side, the installation of a double buffer zone can further reduce the delay.

[0106] For ease of understanding, please also refer to FIG. 10, which is a system architecture diagram provided by an embodiment of the present application. The system architecture diagram in FIG. 10 takes a cloud application as an example, and a cloud application server corresponding to the cloud application may be a cloud game server. As shown in FIG. 10, the system architecture may include a client cluster (including client 1, client 2, ..., client n), a traffic server (including a stream push sub-server and an image recognition sub-server. The stream push sub-server receives an image encoding file uploaded by a client and performs a decoding process, and the image recognition sub-server performs an image recognition process on the decoded image data obtained by decoding by the stream push sub-server), and a cloud game server. The following is a detailed description.

[0107] Client cluster: When each client executes a cloud application (e.g., a cloud game application), it displays the screen of the cloud application (e.g., the screen of the cloud game application). When executing the cloud game application, the user's screen is collected by a camera, encoded, and image data is obtained and uploaded to a stream push sub-server in the traffic server (the stream push sub-server is any one of the servers having a data receiving function and a decoding function, and mainly receives the image encoding file uploaded from the client and performs a decoding process).

[0108] Stream push sub-server: Receives image data uploaded by a client, performs decoding processing, obtains a decoded image having an initial image format (e.g., YUV format), and sends the decoded image to the image recognition sub-server.

[0109] Image recognition sub-server: converts the decoded image from YUV format to RGB format, then recognizes and extracts the user's person image data or human body keypoints in the image, and sends the user's person image or human body keypoints to the cloud game server. Here, the stream push sub-server and the image recognition sub-server jointly constitute a traffic server, so that the traffic server is equipped with image decoding and image recognition functions. Of course, the stream push sub-server and the image recognition sub-server can be independent servers and perform corresponding tasks respectively (i.e., the stream push sub-server receives and decodes image encoding files, and the image recognition sub-server performs image recognition processing on the decoded data). Here, in order to reduce data receiving delay, both the stream push sub-server and the image recognition sub-server can perform frame skip processing.

[0110] Cloud game server: the cloud game server is a cloud application server corresponding to the client, and when the client runs the cloud game application, the cloud game server provides a corresponding computing server. The cloud game server receives user character image data or human body key points, and renders and displays the user character image data. Or, the cloud game server uses the human body key points to operate a virtual animated character in the cloud game application to realize animation (i.e., instead of projecting the user character image to the cloud game application, it operates the virtual animated character to synchronize with the user's actual motion state).

[0111] For ease of understanding, please also refer to Fig. 11, which is a system flow schematic diagram provided by an embodiment of the present application. The flow corresponds to the system architecture corresponding to Fig. 10. As shown in Fig. 11, the flow includes S31 to S36. S31: The client collects camera images. S32: The client encodes the collected images. S33: The stream push subserver decodes the encoded data. S34: The image recognition sub-server converts the decoded image from YUV format to RGB format. S35: The cloud application game server recognizes the user's person image or human body key points. S36: Display a human image or virtual video.

[0112] For specific implementation forms of S31 to S36, please refer to the descriptions of the embodiments corresponding to the above-mentioned Figs. 3, 5, 7, and 8, and they will not be repeated here, and their beneficial effects will not be repeated here either.

[0113] Here, when the client collects an object, if another object is collected at the same time (another user enters the imaging field of view), the client generates object selection prompt information and allows the user to select the final collected object. Or, the client automatically determines the object based on the clarity and occupancy area of ​​the object. For example, if the client collects object 1 and object 2 at the same time, but object 2 is far from the lens and the collected screen is not clear, and object 1 is close to the lens and the collected screen is clear, the client automatically selects object 1 as the final collected object.

[0114] Please refer to Fig. 12, which is an interaction flow chart provided by an embodiment of the present application. The interaction flow may be an interaction flow between a client, a stream push sub-server, an image recognition sub-server, and a cloud application server (take a cloud game server as an example). As shown in Fig. 12, the interaction flow includes at least the following steps S41 to S54. S41: A connection is established between a client and a stream push sub-server. When a user launches a cloud gaming application through a client, a connection is established between the client and the stream push sub-server (e.g., a persistent Websocket connection is established). S42: A connection is established between the image recognition sub-server and the cloud gaming server. When a user launches a cloud gaming application via a client, a connection is established between the cloud gaming server (where the cloud gaming software tool development kit (SDK) is integrated) and the image recognition sub-server (e.g., a Transmission Control Protocol (TCP) connection is established). S43: The cloud gaming server sends a collection notification to the corresponding client. The collection notification may be a message notifying the start of image collection. S44: The client sends a stream push message to the stream push sub-server. The client activates the camera based on the collection notification, and notifies the stream push subserver of the device identifier of the cloud game server, the identifier of the collected user, and the width and height of the image collected by the camera together, so that the stream push subserver prepares to receive the data. The stream push message includes the device identifier of the cloud game server, the identifier of the collected user, and the width and height of the image collected by the camera. S45: The stream push sub-server sends a stream push message to the image recognition sub-server. After receiving the stream push message from the client, the stream push sub-server and the image recognition server establish a TCP connection to send the stream push message to the image recognition sub-server. S46: The client begins to collect and encode images. S47: The client transmits the encoded data to the stream push sub-server. S48: The stream push sub-server performs decryption and obtains the decrypted data. S49: The stream push sub-server sends the decoded data to the image recognition sub-server. S50: The image recognition sub-server performs format conversion on the decoded data and performs image recognition. S51: The image recognition sub-server transmits the recognition data to the cloud gaming server. S52: The cloud gaming server renders the recognition data to obtain rendering data. S53: The cloud gaming server sends the rendering data to the client. For specific implementation forms of S46 to S53, please refer to the description of the embodiments corresponding to the above-mentioned Figs. 3, 5, 7 and 8, and they will not be repeated here, nor will their beneficial effects be repeated.

[0115] Please refer to Figure 13, which is a structural schematic diagram of a data processing device provided by an embodiment of the present application. The data processing device can be a computer program (including program code) executed on a computer device, for example, the data processing device is application software, and the data processing device executes the method of Figure 3. As shown in Figure 13, the data processing device 1 includes: A data acquisition module 11 that acquires first image data transmitted from a first client and stores the first image data in a receiving queue, where the first image data is image data including an acquired object when the first client executes a cloud application; an image recognition module 12 that performs image recognition processing on the first image data in the receiving queue; a queue update module 13 for storing second image data continuously acquired and transmitted from the first client in a receiving queue during image recognition processing of the first image data, and acquiring an updated receiving queue; and an area transmission module 14 that transmits the first object image data included in the first object area to a target cloud application server when a first object area in which the object is located in the first image data is extracted by the image recognition process, the target cloud application server renders the first object image data to obtain rendering data, and transmits the rendering data to the first client; The area sending module 14 further synchronously performs image recognition processing on the second image data having the last received timestamp in the update receiving queue. For specific implementations of the data acquisition module 11, the image recognition module 12, the queue update module 13 and the area transmission module 14, please refer to the description of S101 to S103 in the embodiment corresponding to FIG. 3 above, and the description will not be repeated here.

[0116] In one embodiment, the data acquisition module 11 comprises: An image receiving unit 111 receives first image data sent from a first client, where the first image data is data obtained after the first client performs encoding processing on an original image frame, and the original image frame is an image frame collected when the first client runs a cloud application; The storage unit 112 acquires a reception timestamp at which the first image data is received, and stores the first image data in a reception queue so as to associate the first image data with the reception timestamp. For specific implementations of the image receiving unit 111 and the storage unit 112, please refer to the description of S101 to S103 in the embodiment corresponding to FIG. 3 above, and the description will not be repeated here.

[0117] In one embodiment, the image recognition module 12: a data decoding unit 121 for performing a decoding process on the first image data to obtain decoded image data having an initial image format; a format conversion unit 122 performing format conversion on the decoded image data to obtain an original image frame having a standard image format; an image recognition unit 123 for performing image recognition processing on the original image frames having a standard image format; For specific implementations of the data decoding unit 121, the format conversion unit 122, and the image recognition unit 123, please refer to the description of S101 in the embodiment corresponding to FIG. 3 above, and the description will not be repeated here.

[0118] In one embodiment, the image recognition unit 123 is a keypoint recognition subunit 1231 for recognizing object edge keypoints in an original image frame of the object; a curve connection subunit 1232 for connecting the object edge key points to obtain an object edge curve of the object; and a region determining subunit 1233 for determining an initial object region where the object is located in the original image frame based on the region covered by the object edge curve in the original image frame, and determining a first object region based on the initial object region. For specific implementations of the keypoint recognition subunit 1231, the curve connection subunit 1232 and the region determination subunit 1233, please refer to the description of S102 in the embodiment corresponding to FIG. 3 above, and the description will not be repeated here.

[0119] In one embodiment, the image recognition unit 123 further performs a step of obtaining object recognition placement information for the object and an object recognition placement portion indicated by the object recognition placement information, matching the object recognition placement portion to the object key portion, and determining a first object region based on an initial object region when the object recognition placement portion matches the object key portion; and determining that the first object region cannot be extracted by the image recognition process when the object recognition placement portion does not match the object key portion.

[0120] In one embodiment, the image recognition unit 123 further determines the image data having the first received timestamp in the update receiving queue as the image data to be recognized, performs image recognition processing on the image data to be recognized, and when the image recognition processing extracts an object area to be processed in which an object is located in the image data to be recognized, the area sending module 14 further sends the object area to be processed to the target cloud application server.

[0121] In one embodiment, the image recognition unit 123 specifically includes: Obtain an object key portion of an object to be displayed in the initial object area; If the object key portion has portion integrity, determine the initial object portion as the first object portion; When the object key portion does not have portion integrity, obtain N sample image frames in the sample database, obtain a sample image frame to be processed corresponding to the object from the N sample image frames, and determine a first object region based on the sample image frame to be processed and the initial object region, where N is a positive integer.

[0122] In one embodiment, the image recognition unit 123 specifically includes: Acquire overall part information in the sample image frame to be processed; Based on the object key part, a part region to be fused is determined from the whole part information; The part region to be merged and the initial object region are merged to obtain a first object region.

[0123] In one embodiment, the first image data carries a device identifier to be verified, the device identifier to be verified is a device identifier of a binding cloud application server, and the binding cloud application server and the first client have a binding relationship; The area transmission module 14 is A set obtaining unit 141 for obtaining a set of stored device identifiers, where the set of stored device identifiers includes M stored device identifiers, where one stored device identifier corresponds to one registered cloud application server, where M is a positive integer; an identifier matching unit 142, which determines that the binding cloud application server indicated by the device identifier to be confirmed belongs to the registered cloud application server when a stored device identifier matching the device identifier to be confirmed exists among the M stored device identifiers, determines the binding cloud application server indicated by the device identifier to be confirmed as a target cloud application server, and sends the first object image data to the target cloud application server; For specific implementations of the set obtaining unit 141 and the identifier matching unit 142, please refer to the description of S103 in the embodiment corresponding to FIG. 3 above, and the description will not be repeated here.

[0124] In one embodiment, the data processing device 1 comprises: a registration request receiving module 15 for receiving a registration request sent from a cloud application server to be registered, the registration request being generated after the cloud application server to be registered receives an application start notification sent from a second client, the application start notification being generated in response to an application start operation by the second client on the cloud application; an index detection module 16 for detecting device index information of a cloud application server to be registered based on a registration request; and an identifier addition module 17 that, if the device index information satisfies the processing quality condition, acquires a device identifier to be stored in the cloud application server to be registered, stores the device identifier to be stored in a stored device identifier set, converts the cloud application server to be registered into a registered cloud application server, and converts the device identifier to be stored into the stored device identifier. For specific implementations of the registration request receiving module 15, the index detecting module 16, and the identifier adding module 17, please refer to the description of S103 in the embodiment corresponding to FIG. 3 above, and will not be repeated here.

[0125] In one embodiment, the device characteristic information includes a network quality parameter and a device version; The indicator detection module 16 includes: a parameter obtaining unit 161 for obtaining network quality parameters and device versions of a cloud application server to be registered according to a registration request; an index determining unit 162, which determines that the device index information meets the processing quality condition when the network quality parameter reaches the parameter threshold and the device version matches the quality standard version; The indicator determination unit 162 further determines that the equipment indicator information does not meet the processing quality condition when the network quality parameter does not reach a parameter threshold or the equipment version does not match the quality standard version. For specific implementations of the parameter obtaining unit 161 and the index determining unit 162, please refer to the description of S103 in the embodiment corresponding to FIG. 3 above, and the description will not be repeated here.

[0126] In an embodiment of the present application, when a client (e.g., a first client) acquires first image data including an object, the first image data is sent to a related computer device (e.g., a traffic server), and the traffic server performs image recognition processing, so that image recognition does not need to be performed locally on the client, and the traffic server with high computing power performs image recognition processing on the first image data, thereby improving image recognition efficiency and clarity. In addition, in the present application, the traffic server stores the received first image data in a receiving queue, and in the process of performing image recognition processing on the first image data, continuously synchronously acquires second image data from the first client, stores the second image data in a receiving queue, and acquires an updated receiving queue. That is, when the traffic server in the present application performs image recognition processing on the first image data, it does not suspend the reception of the second image data, and the receiving queue can realize the synchronous progress of image processing and image reception, thereby reducing image transmission delay. Furthermore, when the traffic server extracts a first object region where an object is located in the first image data through image recognition processing, the traffic server transmits the first object image data included in the first object region to the target cloud application server, and the target cloud application server performs rendering and transmits the rendering data to the first client for display in the cloud application. After extracting the first object region, the traffic server can obtain the second image data having the last received timestamp in the receiving queue, and continue to process the second image data.As can be seen from this, in the present application, after performing image recognition processing on certain image data, image data having a last received timestamp is obtained from the receiving queue and processed, and image data is not recognized one by one according to the chronological order of the receiving timestamps, so that the recognition efficiency of image data can be improved, and since the image data having the latest received timestamp is collected based on the current action of the object, even when image recognition and display are performed on the image data having the last received timestamp, it matches the current action of the object. As described above, the present application can improve image recognition efficiency, reduce delays in image transmission, and ensure that the virtual action of a virtual object displayed in a cloud application matches the current action state of the object.

[0127] Please refer to Figure 14, which is a structural schematic diagram of another data processing device provided by an embodiment of the present application. The data processing device can be a computer program (including program code) executed on a computer device, for example, the data processing device is application software, and the data processing device executes the method of Figure 8. As shown in Figure 14, the data processing device 2 includes: an area storage module 21 that receives first object image data transmitted from a traffic server and stores it in a first buffer zone in a buffer zone set whose operating state is in a storage state, the first object image data being image data included in a first object area, the first object area being an area in which an object acquired in the first image data after the traffic server performs an image recognition process on the first image data is located, the first image data being transmitted to the traffic server from a first client, and the area storage module 21 being image data including the object acquired when the first client executes a cloud application; an area rendering module 22 for, when a second buffer zone in the set of buffer zones, whose operational state is a read state, does not contain unprocessed object image data, adjusting the operational state of the first buffer zone to a read state, adjusting the operational state of the second buffer zone to a store state, reading a first object area from the first buffer zone, whose operational state is a read state, and performing a rendering process on the first object area; an area receiving module 23 for receiving second object image data transmitted from a traffic server during rendering of a first object area and storing the second object image data in a second buffer zone whose operating state is in a storage state, the second object image data being image data included in a second object area, the second object area being obtained by the traffic server extracting the first object area and then performing image recognition processing on the second image data, the second object area being an area in which an object is located in the second image data, the second image data being image data having a last reception timestamp obtained from a receiving queue when the traffic server extracts the first object area, the second image data in the receiving queue being continuously obtained from a first client during the traffic server performing image recognition processing on the first image data; and a state adjustment module 24 for, when first rendering data corresponding to the first object image data is acquired, adjusting the operation state of the first buffer zone to a storage state, adjusting the operation state of the second buffer zone to a reading state, reading second object image data from the second buffer zone whose operation state is in the reading state, and performing a rendering process on the second object image data. For specific implementation forms of the area storage module 21, the area rendering module 22, the area receiving module 23 and the state adjustment module 24, please refer to the description of S801 to S804 in the embodiment corresponding to Figure 8 above, and the description will not be repeated here.

[0128] In one embodiment, the condition adjustment module 24 comprises: an identifier obtaining unit 241 for obtaining a read pointer identifier representing a read state corresponding to a first buffer zone and a storage pointer identifier representing a storage state corresponding to a second buffer zone when obtaining first rendering data corresponding to a first object region; an identifier switching unit 242 for switching a read pointer identifier corresponding to a first buffer zone to a store pointer identifier, where the operation state of the first buffer zone having the store pointer identifier is a store state; The identifier switching unit 242 further switches the storage pointer identifier of the second buffer zone to the read pointer identifier, and the operation state of the second buffer zone having the read pointer identifier is the read state. For specific implementations of the identifier acquisition unit 241 and the identifier switching unit 242, please refer to the description of S804 in the embodiment corresponding to FIG. 8 above, and the description will not be repeated here.

[0129] In one embodiment, the data processing device 2 comprises: a buffer zone allocation module 25 for allocating a first buffer zone and a second buffer zone; an identifier setting module 26 for setting an initial pointer identifier of a first buffer zone to a store pointer identifier and setting an initial pointer identifier of a second buffer zone to a read pointer identifier, where the operation state of the first buffer zone having the store pointer identifier is a store state and the operation state of the second buffer zone having the read pointer identifier is a read state; and a set generation module 27 for generating a buffer zone set based on the first buffer zone whose operational state is in a storage state and the second buffer zone whose operational state is in a read state; For specific implementations of the buffer zone allocation module 25, the identifier setting module 26 and the set generation module 27, please refer to the description of S801 in the embodiment corresponding to FIG.

[0130] In the embodiment of the present application, the frame skip processing of the traffic server can reduce the image reception delay on the traffic server side and improve the image recognition efficiency, and the double buffer zone allocation processing of the target cloud application server can eliminate the need to copy data, just switch the working state of the two buffer zones, and eliminate the need to allocate the buffer zone every time, and the data reception and data processing (e.g., reading and rendering the data) can be performed simultaneously, without waiting for each other, and the delay can be reduced. That is, in reducing the delay on the traffic server side, the installation of the double buffer zone can further reduce the delay.

[0131] Referring to FIG. 15, FIG. 15 is a structural schematic diagram of a computer device provided by an embodiment of the present application. As shown in FIG. 15, the device 1 of the embodiment corresponding to the above FIG. 13 or the device 2 of the embodiment corresponding to the above FIG. 14 may be applied to the above computer device 8000, and the above computer device 8000 may include a processor 8001, a network interface 8004 and a memory 8005, and may further include a user interface 8003 and at least one communication bus 8002. The communication bus 8002 realizes the connection communication between these components. The user interface 8003 may include a display and a keyboard, and may further include a standard wired interface and a wireless interface. Preferably, the network interface 8004 may include a standard wired interface and a wireless interface (e.g., a WI-FI interface). The memory 8005 may be a high-speed RAM memory, or may be a non-volatile memory, such as at least one magnetic disk memory. Preferably, the memory 8005 may further be at least one storage device remote from the processor 8001. As shown in FIG. 15, the memory 8005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0132] In the computer equipment 8000 of Figure 15, the network interface 8004 can provide network communication functions, the user interface 8003 mainly provides an input interface for the user, and the processor 8001 calls the equipment control application program stored in the memory 8005 to realize the data processing method provided by the above embodiment.

[0133] Here, the computer device 8000 described in the embodiments of the present application may execute the description of the data processing method of the embodiment corresponding to Figures 3 to 8 above, or may execute the description of the data processing device 1 of the embodiment corresponding to Figure 13 above, or the description of the data processing device 2 of the embodiment corresponding to Figure 14 above, which will not be repeated here. Also, the description of the beneficial effects of the same methods will not be repeated.

[0134] In addition, the embodiment of the present application further provides a computer-readable storage medium, in which a computer program executed by the computer device 8000 of the data processing is stored, the computer program includes program instructions, and the processor executes the program instructions to execute the above data processing method of the embodiment corresponding to Figures 3 to 8, which will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the embodiment of the computer-readable storage medium of the present application, please refer to the description of the method embodiment of the present application.

[0135] The computer-readable storage medium may be the data processing device provided by any one of the above embodiments, or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may be an external storage device of the computer device, such as a connected hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., disposed in the computer device. Moreover, the computer-readable storage medium includes the internal storage unit of the computer device and also includes an external storage device. The computer-readable storage medium stores the computer program, other programs and data required by the computer device. The computer-readable storage medium may also temporarily store data that has been output or is to be output.

[0136] According to one aspect of the present application, there is provided a computer program product or computer program, the computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computing device reading the computer instructions from the computer readable storage medium, and the processor executing the computer instructions, thereby causing the computing device to perform a method provided by any one of the aspects of the embodiments of the present application.

[0137] The terms "first", "second", etc. in the description, claims and drawings of the embodiments of this application are not intended to describe a particular order, but to distinguish different objects. Also, the term "comprises" and any variations thereof are intended to include a non-exclusive inclusion. For example, a process, method, apparatus, product or device comprising a series of steps or units is not limited to the recited steps or modules, and preferably includes unrecited steps or modules, or further includes other steps or units inherent in such process, method, apparatus, product or device.

[0138] As can be appreciated by those skilled in the art, each exemplary unit and algorithm step described in conjunction with the embodiments disclosed herein may be realized by electronic hardware, computer software, or a combination of both, and in the above description, each exemplary configuration and step is generally described according to function in order to clearly describe the compatibility between hardware and software. Whether these functions are executed in a hardware form or in a software form depends on the specific application and design constraints of the technical solution. Those skilled in the art may realize the described functions using different methods for each specific application, and such realization does not go beyond the scope of this application.

[0139] The method and related apparatus provided by the embodiments of the present application are described with reference to the method flowcharts and / or structural diagrams provided by the embodiments of the present application, and in particular, computer-readable instructions are provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, and the instructions executed by the processor of the computer or other programmable data processing device generate an apparatus that realizes the function specified by the flow or flows of the flowcharts and / or the block or blocks of the structural diagrams. The computer-readable instructions are further stored in a computer-readable memory that guides the computer or other programmable data processing device to operate in a specific manner, and the instructions stored in the computer-readable memory generate a product including an instruction apparatus, and the instruction apparatus realizes the function specified by the flow or flows of the flowcharts and / or the block or blocks of the structural diagrams. These computer-readable instructions may further be loaded into a computer or other programmable data processing device to cause the computer or other programmable device to execute a series of operational steps to generate a computer-implemented process, the instructions executing on the computer or other programmable device providing steps to implement the functions specified by a flow or flows in the flowcharts and / or a block or blocks in the structural schematic diagrams.

[0140] The above disclosure does not limit the scope of the claims of this application, but is merely a preferred embodiment of this application, and any equivalent modifications made based on the claims of this application still belong to the scope of coverage of this application.

Claims

1. 1. A data processing method implemented by a computing device, comprising: A step of acquiring first image data transmitted from a first client and storing the first image data in a receiving queue, the first image data being image data including an object acquired when the first client executes a cloud application; a step of performing an image recognition process on the first image data in the receiving queue, and continuously acquiring second image data from the first client during the image recognition process of the first image data and storing the second image data in the receiving queue to acquire an updated receiving queue, wherein the image recognition process on the first image data and the acquisition and storage of the second image data are performed synchronously; a step of, when a first object area in which the object is located in the first image data is extracted by the image recognition process, sending the first object image data included in the first object area to a target cloud application server, and performing image recognition processing on second image data having a latest reception timestamp in the update reception queue, in which the target cloud application server renders the first object image data to obtain rendering data, and transmits the rendering data to the first client; The method includes:

2. The step of acquiring the first image data transmitted from the first client and storing it in a receiving queue includes: receiving the first image data transmitted from the first client, the first image data being data acquired by the first client performing an encoding process on an original image frame, the original image frame being an image frame collected by the first client during execution of the cloud application; acquiring a reception timestamp indicating when the first image data was received, and storing the first image data and the reception timestamp in the reception queue in association with each other; 2. The method of claim 1, comprising:

3. The step of performing image recognition processing on the first image data in the receiving queue includes: performing a decoding process on the first image data to obtain decoded image data having an initial image format; performing a format conversion on the decoded image data to obtain the original image frame having a standard image format; performing image recognition processing on the original image frames having the standard image format; 3. The method of claim 2 comprising:

4. The step of performing image recognition processing on the original image frame having the standard image format includes: recognizing object edge keypoints in the original image frame of the object; connecting the object edge keypoints to obtain an object edge curve of the object; determining an area covered by the object edge curve in the original image frame as an initial object area where the object is located in the original image frame; determining the first object region based on the initial object region; 4. The method of claim 3, comprising:

5. Before determining the first object region based on the initial object region, the method further comprises: obtaining an object key portion of the object represented by the initial object region; obtaining object recognition placement information for the object and an object recognition placement portion indicated by the object recognition placement information, and matching the object recognition placement portion with the object key portion; performing a step of determining the first object region based on the initial object region when the object recognition placement portion matches the object key portion; determining that the first object region cannot be extracted by image recognition processing when the object recognition placement portion does not match the object key portion; The method of claim 4 further comprising:

6. After determining that the first object region cannot be extracted by an image recognition process when the object recognition placement portion does not match the object key portion, the method further comprises: determining image data having a first reception timestamp in the update reception queue as image data to be recognized; performing an image recognition process on the image data to be recognized, and when an object area to be processed in which the object is located in the image data to be recognized is extracted by the image recognition process, transmitting the object area to be processed to the target cloud application server; The method of claim 5 further comprising:

7. The step of determining the first object region based on the initial object region comprises: obtaining an object key portion of the object represented by the initial object region; determining the initial object region as the first object region if the object key region has region integrity; when the object key portion does not have portion integrity, obtaining N sample image frames in a sample database, obtaining a sample image frame to be processed corresponding to the object from the N sample image frames, and determining the first object region based on the sample image frame to be processed and the initial object region, where N is a positive integer; 5. The method of claim 4, comprising:

8. determining the first object region based on the sample image frame to be processed and the initial object region, acquiring overall part information of the sample image frame to be processed; determining a part region to be a fusion target from the whole part information based on the object key part; fusing the part region to be merged with the initial object region to obtain the first object region; 8. The method of claim 7, comprising:

9. The first image data carries a device identifier to be confirmed, the device identifier to be confirmed is a device identifier of a binding cloud application server, and the binding cloud application server and the first client have a binding relationship; The step of transmitting the first object image data included in the first object region to a target cloud application server includes: obtaining a set of stored device identifiers, the set of stored device identifiers including M stored device identifiers, one stored device identifier corresponding to one registered cloud application server, where M is a positive integer; if there is a stored device identifier that matches the device identifier to be confirmed among the M stored device identifiers, determining that the binding cloud application server indicated by the device identifier to be confirmed belongs to a registered cloud application server, determining the binding cloud application server indicated by the device identifier to be confirmed as the target cloud application server, and transmitting the first object image data included in the first object area to the target cloud application server; 2. The method of claim 1, comprising:

10. The method comprises: receiving a registration request sent from a cloud application server to be registered, the registration request being generated by the cloud application server to be registered after receiving an application launch notification sent from a second client, the application launch notification being generated by the second client in response to an application launch operation on the cloud application; detecting device index information of the cloud application server to be registered based on the registration request; if the device index information satisfies a processing quality condition, acquiring a device identifier to be stored in the cloud application server to be registered, storing the device identifier to be stored in the stored device identifier set, converting the cloud application server to be registered into a registered cloud application server, and converting the device identifier to be stored into a stored device identifier; The method of claim 9 further comprising:

11. The device indicator information includes a network quality parameter and a device version; The step of detecting device index information of the cloud application server to be registered based on the registration request includes: obtaining network quality parameters and a device version of the cloud application server to be registered according to the registration request; determining that the device index information satisfies the processing quality condition when the network quality parameter reaches a parameter threshold and the device version matches a quality standard version; determining that the device index information does not satisfy the processing quality condition when the network quality parameter does not reach the parameter threshold or the device version does not match the quality standard version; 11. The method of claim 10, comprising:

12. The step of performing image recognition processing on second image data having a latest reception timestamp in the update reception queue includes: obtaining the second image data having the latest received timestamp in the update receive queue; performing an image recognition process on the second image data and deleting historical image data in the update receiving queue, the historical image data being image data in the update receiving queue having an earlier reception timestamp than the second image data; 2. The method of claim 1, comprising:

13. 1. A data processing method implemented by a computing device, comprising: A step of receiving first object image data transmitted from a traffic server and storing the first object image data in a first buffer zone in a buffer zone set whose operating state is in a storage state, the first object image data being image data included in a first object area, the first object area being an area in which an object acquired in the first image data after the traffic server performs an image recognition process on the first image data is located, the first image data being data transmitted from a first client to the traffic server, and the first image data being image data including the object acquired when the first client executes a cloud application; if a second buffer zone in the set of buffer zones, the operational state of which is in the read state, does not contain unprocessed object image data, adjusting the operational state of the first buffer zone to the read state, adjusting the operational state of the second buffer zone to the store state, reading the first object image data from the first buffer zone, the operational state of which is in the read state, and performing a rendering process on the first object image data; receiving second object image data transmitted from the traffic server during the rendering process of the first object image data, and storing the second object image data in the second buffer zone whose operating state is the storage state, the second object image data being image data included in a second object area, the second object area being obtained by the traffic server extracting the first object area and then performing image recognition processing on the second image data, the second object area being an area in which the object is located in the second image data, the second image data being image data having a latest reception timestamp obtained from an update receiving queue when the traffic server extracts the first object area, and the second image data in the update receiving queue being data continuously obtained from the first client during the process in which the traffic server performs image recognition processing on the first image data; when rendering data corresponding to the first object image data is acquired, adjusting the operation state of the first buffer zone to the storage state, adjusting the operation state of the second buffer zone to the reading state, reading the second object image data from the second buffer zone whose operation state is the reading state, and performing a rendering process on the second object image data; The method includes:

14. When rendering data corresponding to the first object image data is obtained, the step of adjusting the operation state of the first buffer zone to the storage state and adjusting the operation state of the second buffer zone to the reading state includes: when rendering data corresponding to the first object image data is obtained, obtaining a read pointer identifier corresponding to the first buffer zone and representing the read state, and a store pointer identifier corresponding to the second buffer zone and representing the store state; switching a read pointer identifier corresponding to the first buffer zone to the store pointer identifier, wherein an operational state of the first buffer zone having the store pointer identifier is the store state; switching the store pointer identifier of the second buffer zone to the read pointer identifier, the operational state of the second buffer zone having the read pointer identifier being the read state; 14. The method of claim 13, comprising:

15. The method comprises: allocating a first buffer zone and a second buffer zone; setting an initial pointer identifier of the first buffer zone to a store pointer identifier and an initial pointer identifier of the second buffer zone to a read pointer identifier, wherein an operational state of the first buffer zone having the store pointer identifier is the store state and an operational state of the second buffer zone having the read pointer identifier is the read state; generating the set of buffer zones based on the first buffer zone having an operational state in the storage state and a second buffer zone having an operational state in the read state; The method of claim 13 further comprising:

16. A data processing device arranged in a computer device, comprising: A data acquisition module that acquires first image data transmitted from a first client and stores the first image data in a receiving queue, the first image data being image data including an object acquired when the first client executes a cloud application; an image recognition module that performs image recognition processing on the first image data in the receiving queue; a queue update module that continuously acquires second image data from the first client and stores it in the receiving queue to acquire an updated receiving queue during an image recognition process of the first image data, the image recognition process for the first image data and the acquisition and storage of the second image data are performed synchronously with the queue update module; and an area transmission module that transmits the first object image data included in the first object area to a target cloud application server when a first object area in which the object is located in the first image data is extracted by the image recognition process; The area sending module further performs image recognition processing on second image data having a latest received timestamp in the update receiving queue, and the target cloud application server renders the first object image data to obtain rendering data, and sends the rendering data to the first client.

17. A data processing device arranged in a computer device, comprising: an area storage module that receives first object image data transmitted from a traffic server and stores the first object image data in a first buffer zone in a buffer zone set whose operating state is in a storage state, the first object image data being image data included in a first object area, the first object area being an area in which an object acquired by the traffic server performing image recognition processing on the first image data is located in the first image data, and the first image data being image data transmitted from a first client to the traffic server and including an object acquired when the first client executes a cloud application; an area rendering module for, when a second buffer zone in the set of buffer zones, the operational state of which is in a read state, does not contain unprocessed object image data, adjusting an operational state of a first buffer zone to a read state, adjusting an operational state of the second buffer zone to a store state, reading the first object area from the first buffer zone, the operational state of which is in a read state, and performing a rendering process on the first object area; an area receiving module that receives second object image data transmitted from the traffic server during the rendering process of the first object area and stores the second object image data in the second buffer zone in a storage state, the second object image data being image data included in a second object area, the second object area being obtained by the traffic server extracting the first object area and then performing an image recognition process on the second image data, the second object area being an area where an object is located in the second image data, the second image data being image data having a latest reception timestamp obtained from an update receiving queue when the traffic server extracts the first object area, and the second image data in the update receiving queue being data continuously obtained from the first client during the process of the traffic server performing an image recognition process on the first image data; a state adjustment module for, when rendering data corresponding to the first object image data is acquired, adjusting an operation state of the first buffer zone to a storage state, adjusting an operation state of the second buffer zone to a reading state, reading the second object image data from the second buffer zone whose operation state is the reading state, and performing a rendering process on the second object image data; An apparatus comprising:

18. A computer device comprising a processor, a memory and a network interface, A computer device, wherein the processor, the memory, and the network interface are connected, the network interface provides a network communication function, the memory stores program code, and the processor calls the program code to cause the computer device to execute the method according to any one of claims 1 to 15.

19. A computer program comprising computer instructions stored in a computer readable storage medium and which, when read and executed by a processor, causes the processor to perform the method of any one of claims 1 to 15.

Images