Virtual object identification method, device, apparatus and computer readable storage medium

Abstract

The application discloses a virtual object identification method and device, equipment and a computer readable storage medium, and belongs to the computer vision technology. A to-be-identified video can be acquired, and a virtual object to be identified in the to-be-identified video is determined; a first image and a second image of adjacent frames are extracted from the to-be-identified video, and the first image contains a detection box of the virtual object; the first image and the second image are input into a target model to obtain a target detection box of the virtual object in the second image; the target model performs enhanced representation and fusion on a region feature of the virtual object in the detection box in the first image and an object color feature of the second image, so as to predict the target detection box of the virtual object in the second image according to an aggregated feature matrix after fusion; and the virtual object in the to-be-identified video is identified according to the target detection box in the second image. In this way, the accuracy of identifying the virtual object in the video can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a method, apparatus, device, and computer-readable storage medium for identifying virtual objects. Background Technology

[0002] With the development of network technology, various game development platforms have launched a wide variety of games, bringing users a high level of entertainment. However, in game judgment mechanisms, such as judging the actions of virtual objects, victory judgments, death judgments, and boundary crossing judgments, it is generally necessary to consider the trajectory and position of virtual objects in the game world. Therefore, it is necessary to first identify virtual objects in the game screen, and then determine their trajectory and position. Related technologies mainly identify virtual objects by calculating the image brightness difference between adjacent game frames and then using binarization to identify virtual objects in the game screen.

[0003] In the process of researching and practicing existing technologies, the inventors of this application discovered that when identifying virtual objects, the variability of virtual objects in the game screen affects the accuracy of identification, such as the tendency to miss or fail to identify virtual objects in the game screen, resulting in low efficiency and accuracy in identifying virtual objects. Summary of the Invention

[0004] This application provides a virtual object recognition method, apparatus, device, and computer-readable storage medium, which can sequentially identify and accurately label virtual objects from each image frame, thereby improving the accuracy of virtual object recognition.

[0005] This application provides a virtual object recognition method, including:

[0006] Acquire the video to be identified and determine the virtual object to be identified in the video;

[0007] Extract a first image and a second image of adjacent frames from the video to be identified, wherein the first image contains the detection box of the virtual object;

[0008] Input the first image and the second image into the target model to obtain the target detection box of the virtual object in the second image;

[0009] The target model enhances and fuses the region features of the virtual object within the detection box in the first image and the object color features in the second image to predict the target detection box of the virtual object in the second image based on the fused aggregated feature matrix.

[0010] Based on the target detection box in the second image, the virtual objects in the video to be identified are identified.

[0011] Accordingly, embodiments of this application provide a virtual object recognition device, including:

[0012] An acquisition unit is used to acquire a video to be identified and to determine the virtual object to be identified in the video.

[0013] An extraction unit is used to extract a first image and a second image of adjacent frames from the video to be identified, wherein the first image contains the detection box of the virtual object;

[0014] The recognition unit is used to input the first image and the second image into the target model to obtain the target detection box of the virtual object in the second image;

[0015] The target model enhances and fuses the region features of the virtual object within the detection box in the first image and the object color features in the second image to predict the target detection box of the virtual object in the second image based on the fused feature matrix.

[0016] The recognition unit is used to recognize virtual objects in the video to be recognized based on the target detection box in the second image.

[0017] In some embodiments, the identification unit is further configured to:

[0018] Input the first image and the second image into the target model;

[0019] The region features of virtual objects are extracted from the first image using a spatial attention mechanism to obtain a region feature matrix.

[0020] The color feature matrix is ​​obtained by extracting object color features from the second image using a spatial attention mechanism.

[0021] The region feature matrix and the color feature matrix are represented with channel attention enhancement and then fused to obtain the aggregated feature matrix;

[0022] Based on the aggregated feature matrix, the location regions of virtual objects in the second image are labeled to obtain target detection boxes.

[0023] In some embodiments, the identification unit is further configured to:

[0024] The second image is convolved to obtain the feature matrix of the second image;

[0025] Based on the spatial attention mechanism, color feature weights are determined from the second image feature matrix, and the object color features in the second image feature matrix are enhanced by attention according to the color feature weights to obtain the color feature matrix.

[0026] In some embodiments, the identification unit is further configured to:

[0027] Based on the spatial attention mechanism, the initial color feature weights are determined from the color features in the second image feature matrix;

[0028] Based on the initial color feature weights, the object color features in the second image feature matrix are initially enhanced to obtain an initial fusion feature matrix;

[0029] The initial fusion feature matrix is ​​convolved to obtain the initial color feature matrix;

[0030] The spatial attention mechanism is used to perform perceptual processing on the color features in the initial color feature matrix, and the object color features in the initial fusion feature matrix are enhanced based on the target color feature weights obtained from the perceptual processing to obtain the target fusion feature matrix.

[0031] The target fusion feature matrix is ​​convolved to obtain the color feature matrix.

[0032] In some embodiments, the identification unit is further configured to:

[0033] Based on the spatial attention mechanism, the second image feature matrix is ​​convolved to obtain the first convolutional feature matrix;

[0034] The first convolutional feature matrix is ​​subjected to eigenvalue normalization, and the eigenvalues ​​in the initial eigenvalue matrix obtained by normalization are subjected to local response normalization to obtain the target eigenvalue matrix.

[0035] The target feature value matrix is ​​convolved to obtain the second convolution feature matrix;

[0036] Color features in the second convolutional feature matrix are classified by color weights to obtain initial color feature weights.

[0037] In some embodiments, the identification unit is further configured to:

[0038] The region feature matrix is ​​perceptually processed based on the channel attention mechanism to obtain the first channel weight, and the region feature matrix is ​​then represented by channel attention enhancement based on the first channel weight to obtain the first channel feature matrix.

[0039] Based on the channel attention mechanism, the color feature matrix is ​​perceptually processed to obtain the second channel weights, and the color feature matrix is ​​then represented by channel attention enhancement based on the second channel weights to obtain the second channel feature matrix.

[0040] The first channel feature matrix and the second channel feature matrix are convolved respectively to obtain the target region feature matrix and the target color feature matrix.

[0041] The target region feature matrix and the target color feature matrix are fused together in the channel to obtain the aggregated feature matrix.

[0042] In some embodiments, the identification unit is further configured to:

[0043] Based on the channel attention mechanism, the color feature matrix is ​​processed by a fully connected layer to obtain the first feature matrix;

[0044] According to the feature compliance, some feature values ​​in the first feature matrix are hidden to obtain the second feature matrix;

[0045] The second channel weights are obtained by integrating the color weights of the second feature matrix through an activation function.

[0046] In some embodiments, the virtual object recognition device further includes a determining unit, used for:

[0047] The aggregated feature matrix is ​​fully connected to obtain the target feature matrix.

[0048] The target feature matrix is ​​subjected to binary classification to obtain the object modality value;

[0049] Based on the object modality value, the location region of the corresponding virtual object in the second image is marked to obtain the target detection box.

[0050] In some embodiments, the identification unit is further configured to:

[0051] Obtain the feature hiding parameters of the target model in the channel attention-enhanced representation, and calculate the feature classification weights based on the feature hiding parameters;

[0052] The target feature matrix is ​​weighted according to the feature classification weights to obtain the target weight feature matrix;

[0053] The matrix difference between the target weight feature matrix and the constant bias matrix is ​​accumulated to obtain the object modality value.

[0054] In some embodiments, the identification unit is further configured to:

[0055] The aggregated feature matrix is ​​fully connected to obtain a target feature matrix containing features of multiple virtual objects;

[0056] The target feature matrix is ​​subjected to binary classification processing by different binary classification modules in the target model to obtain the object modality value corresponding to each virtual object feature;

[0057] Based on the object modality value, the location region of each virtual object in the second image is labeled to obtain the target detection box corresponding to each virtual object.

[0058] In some embodiments, the virtual object recognition device further includes a training unit for:

[0059] Obtain a first sample image and a second sample image representing the relationship between adjacent frames. The first sample image and the second sample image contain the same virtual sample object. The first sample image contains the initial sample detection box of the virtual sample object.

[0060] The first and second sample images are used as input, and the initial sample detection boxes are used as training labels to train the preset model.

[0061] The training of the target model continues until the predicted detection box in the second sample image output by the preset model matches the virtual sample object labeled in the initial sample detection box.

[0062] In some embodiments, the extraction unit is further configured to:

[0063] Extract the first frame image containing the virtual object to be identified from the video to be identified;

[0064] The virtual objects to be identified in the first frame image are labeled to obtain a target image frame containing the detection boxes labeled with the virtual objects;

[0065] The extraction unit is further configured to use the target image frame as the previous frame image and determine the previous frame image as the first image;

[0066] Based on the adjacent frame relationship in the video frame to be identified, the next image frame adjacent to the first image is obtained, and the next image frame is determined as the second image.

[0067] Furthermore, embodiments of this application also provide a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in any of the virtual object recognition methods provided in embodiments of this application.

[0068] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the virtual object recognition methods provided in embodiments of this application.

[0069] Furthermore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed, implement the steps in any virtual object recognition method provided in embodiments of this application.

[0070] This application embodiment can acquire a video to be identified and determine the virtual object to be identified in the video; extract a first image and a second image from adjacent frames of the video to be identified, the first image containing the detection box of the virtual object; input the first image and the second image into a target model to obtain the target detection box of the virtual object in the second image; wherein, the target model enhances and fuses the region features of the virtual object in the detection box in the first image and the object color features in the second image, so as to predict the target detection box of the virtual object in the second image based on the fused aggregated feature matrix; and identify the virtual object in the video to be identified based on the target detection box in the second image. Therefore, this scheme can identify the virtual object to be tracked and identified in the video to be identified, and extract a first image to determine the location of the virtual object and a second image to be identified based on the relationship between adjacent frames from the video to be identified. The first and second images are then enhanced with feature representations in the spatial and channel dimensions using a target model to highlight the shape of the virtual object in the image, making the virtual object more prominent relative to other background elements. Furthermore, the virtual object features extracted from the first and second images are fused, and the virtual object to be identified in the second image is determined based on the aggregated feature matrix. Target detection boxes are then generated to identify the virtual object based on the motion trajectory corresponding to a series of target detection boxes. In this way, the same virtual object can be identified and labeled sequentially from each image frame, effectively avoiding the influence of occlusion or variability of the virtual object in the image on the recognition process, and improving the efficiency and accuracy of virtual object recognition. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a schematic diagram of a virtual object recognition system provided in an embodiment of this application;

[0073] Figure 2 This is a flowchart illustrating the steps of the virtual object recognition method provided in the embodiments of this application;

[0074] Figure 3This is a schematic diagram of another step in the virtual object recognition method provided in the embodiments of this application;

[0075] Figure 4 This is a schematic diagram of the architecture of the virtual object recognition system provided in the embodiments of this application;

[0076] Figure 5 This is a flowchart illustrating the steps of the virtual player character recognition method in game videos provided in this application embodiment;

[0077] Figure 6 This is a schematic diagram of the interface of a game screen including a target detection box, provided in an embodiment of this application;

[0078] Figure 7 This is a schematic diagram of the target model structure in the virtual object recognition method provided in this application embodiment;

[0079] Figure 8 This is a schematic diagram of the spatial perception module provided in an embodiment of this application;

[0080] Figure 9 This is a schematic diagram of the structure of the channel sensing module provided in an embodiment of this application;

[0081] Figure 10 This is a schematic diagram of the structure of the virtual object recognition device provided in the embodiments of this application;

[0082] Figure 11 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0083] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0084] This application provides a virtual object recognition method, apparatus, device, and computer-readable storage medium. Specifically, this application will describe the virtual object recognition apparatus from the perspective of the virtual object recognition apparatus. This virtual object recognition apparatus can be integrated into a computer device, which can be a server or a user terminal, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The user terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart home appliance, vehicle terminal, smart voice interaction device, aircraft, etc., but is not limited to these.

[0085] It is understood that in the specific implementation of this application, data related to user information, user usage records, and user status are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0086] The virtual object recognition method provided in this application can be applied to various scenarios of virtual object recognition. These scenarios are not limited to those implemented through cloud services, big data, artificial intelligence, or a combination thereof. The following embodiments illustrate these methods:

[0087] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0088] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0089] See Figure 1 This is a schematic diagram of a scenario for the virtual object recognition system provided in this application embodiment. The virtual object recognition system can be applied to scenarios involving the tracking and recognition of the same virtual object between any adjacent image frames, such as tracking and recognizing virtual objects in a game world in the game background, or tracking and recognizing virtual objects in a "metaverse." Furthermore, it can also be applied to tracking and recognizing virtual objects in videos (such as movies), etc. The virtual object recognition system may include a terminal and a server.

[0090] For example, the terminal is used to provide videos requiring virtual object recognition and object recognition tasks, and the server is used to recognize virtual objects in the videos or images. Specifically, the server is used to acquire the video to be recognized and determine the virtual objects to be recognized in the video; extract a first image and a second image from adjacent frames of the video to be recognized, the first image containing the detection box of the virtual object; input the first image and the second image into a target model to obtain the target detection box of the virtual object in the second image; wherein, the target model enhances and fuses the region features of the virtual object within the detection box in the first image and the object color features of the second image, so as to predict the target detection box of the virtual object in the second image based on the fused aggregated feature matrix; and recognize the virtual object in the video to be recognized based on the target detection box in the second image.

[0091] In addition, the system may also include a terminal, on which a client for virtual object recognition function is installed. The terminal can implement the virtual object recognition method of the present application embodiment through the client. The process of implementing the virtual object recognition method on the client is the same as the execution process of the server described above, and will not be described in detail here.

[0092] The virtual object recognition process may include the following steps: determining the virtual object to be recognized, extracting the first and second images from adjacent frames, labeling the virtual object in the second image, and recognizing the virtual object.

[0093] Example 1, taking the identification of virtual objects in a game screen as an example, can be a game character, whose attribute categories include virtual player characters and non-player characters (NPCs). Taking a virtual player character as an example, specifically, the game video and object recognition task corresponding to the game are first obtained, and the virtual player character to be identified in the game video is determined based on the object recognition task; then, the virtual player character to be tracked and identified is locked at the beginning of the game video, or when the virtual player character to be identified first appears in the game video screen, such as by marking the virtual player character with relevant markers (such as detection boxes of various shapes such as fingers, arrows, rectangles, circles, etc.); then, the first image frame in which the virtual player character appears is taken as the first identified image, and the next images with adjacent frame relationships are obtained sequentially according to the temporal relationship of the video frames. The system identifies and labels virtual player characters in each subsequent image frame. After identification, it continues to identify virtual player characters in the next image frame until the virtual player character to be identified ends in the next image frame. In this way, it can identify virtual player characters between image frames with temporal relationships, and track and identify virtual player characters in the video that are in motion or not, until the virtual player character disappears in the next image frame or until the virtual player character is in the last image frame at the end of the time sequence, so as to track and identify them based on the labels.

[0094] Example 2, taking the identification of virtual objects in a movie video as an example, the virtual objects to be identified are the characters in the movie. Specifically, firstly, the server in the system can acquire the object recognition task and the movie video, and determine the character to be recognized based on the object recognition task; then, it locks the character to be tracked and recognized at the beginning of the movie video, or when the character to be recognized first appears on the movie screen, such as by marking the character with a detection box; then, taking the image frame where the character first appears as the first recognized image, according to the temporal relationship of the video frames, it sequentially acquires the next image frame with an adjacent frame relationship, and identifies the character in the next image frame, marking it with a detection box. After detecting virtual objects, it continues to identify the character in the "next image frame" until the character to be recognized ends in the "next image frame"; in this way, it realizes the identification of characters between image frames with temporal relationships, and realizes the tracking and recognition of characters in the movie video who are in a state of displacement or not in a state of displacement, until the character ends in the last image frame at the end of the temporal sequence, thus completing the identification and tracking of characters in the movie video.

[0095] The following sections provide detailed descriptions of each example. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0096] In this embodiment, the description will focus on a virtual object recognition device, which can be integrated into a computer device such as a server. See also Figure 2 , Figure 2 This is a flowchart illustrating the steps of a virtual object recognition method provided in this application embodiment. Taking the virtual object recognition device specifically integrated on a server as an example, the specific process when the processor on the server executes the program instructions corresponding to the virtual object recognition method is as follows:

[0097] 101. Obtain the video to be recognized and identify the virtual objects to be recognized in the video.

[0098] In this embodiment of the application, in order to track and identify virtual objects in consecutive image frames of a video, it is necessary to identify virtual objects in each image frame sequentially, and mark or annotate them with detection boxes when they are identified, so that the virtual objects can be tracked and identified in the future according to the temporal relationship of the video frames and the multiple annotated detection boxes.

[0099] However, in the process of recognizing virtual objects, the object recognition task and the video to be recognized can be obtained first, so as to determine the virtual objects to be recognized in the video based on the object recognition task.

[0100] The object recognition task can be a task for determining the virtual object to be recognized, which may include the object information of the virtual object to be recognized; wherein, the object information may be the identifier, name, object attributes (such as appearance attributes) of the virtual object, etc., used to distinguish and determine the virtual object to be recognized, such as determining the virtual object to be recognized in a video or the first frame image based on the object information.

[0101] The video to be identified can be an existing video or a video being recorded, and all or part of the image frames in the video contain virtual objects. It should be noted that when identifying virtual objects later, the existing video to be identified can be segmented into frames to obtain multiple image frames; in addition, for the video being recorded, each image frame can be captured in real time.

[0102] Example 1: To score a player's gameplay, the game backend can set up an object recognition task for that player's account in a specific game match. This task could be based on the object information of the virtual player character used by the player in the current match. Then, the official recorded video of the current game match (recorded by the game platform) is obtained as the video to be recognized. This allows the identification of the virtual player character in the game match based on the object information in the object recognition task. Here, the virtual player character can be understood as a game character in the game world. It should be noted that this embodiment of the application is not limited to match-based games when identifying virtual player characters in the game world; it can also be applied to other types of game scenarios.

[0103] Example 2: In order to score virtual objects in a movie video, such as acting skills, frequency of appearance, and movement trajectory, the movie video is used as the video to be identified. The virtual objects in the movie video are then identified in order to track and identify them. Therefore, it is necessary to first set up an object recognition task so as to determine the virtual objects to be identified in the movie video based on the object information in the object recognition task. The virtual objects can be understood as the characters in the movie video.

[0104] Example 3: In the "Metaverse World" scenario, in order to track and identify virtual objects, the background of the "Metaverse World" can obtain the image stream, image frames or videos presented in the interface of the playback device (the device providing the vision) of the "Metaverse World" as the video to be identified, and set an object recognition task so as to determine the virtual objects that need to be identified in the "Metaverse World" based on the object information in the object recognition task. The virtual objects can be objects in the form of characters participating in the "Metaverse World", virtual world characters, etc., which are not limited here.

[0105] The virtual object can be an object to be identified in an image frame, specifically a character to be tracked and identified in a video. This virtual object can be in a state of motion or not; there is no limitation here. It should be noted that the meaning of the virtual object can be determined according to its context, such as a game character in a game world, a script character in a movie, a virtual character in a "metaverse," or a character in other scenarios; there is no limitation here.

[0106] To identify virtual objects in a video frame, a relevant object recognition task must first be obtained. For example, using a video platform, personnel can select the video to be recognized through a client application and set the object information for the virtual objects to be tracked and identified. This information can be entered and selected in the corresponding object task setting interface, such as the virtual object's identifier, number, name, etc., so that an object recognition task can be generated in the background. The server then obtains the object recognition task and the video to be recognized to determine the virtual objects to be tracked and identified within the video.

[0107] By using the above methods, the virtual objects that need to be identified in the video can be determined, so that the virtual objects can be identified from a single or multiple consecutive image frames / images, thereby achieving the tracking and identification of virtual objects.

[0108] It should be noted that the tracking and recognition of virtual objects is primarily applied in video or image stream scenarios. By sequentially identifying and labeling virtual objects in each image or frame, the same virtual object can be tracked and identified throughout the entire video or image stream. Therefore, this recognition process is a gradual one, consisting of multiple recognition rounds. For example, identifying a virtual object in an image frame and marking it with a detection box is considered one recognition round.

[0109] 102. Extract the first and second images of adjacent frames from the video to be identified.

[0110] In this embodiment of the application, in each frame recognition round, it is necessary to combine the previous image frame whose virtual object position has been determined to guide the determination of the virtual object position in the current image frame and mark it, thereby completing the recognition of the virtual object.

[0111] Specifically, in the process of identifying virtual objects in each image frame, the previous image frame in which the virtual object has been identified can be used as the first image of the current identification round as a basis, and the current image frame adjacent to the first image can be used as the second image to be identified, so that the virtual object features in the first image and the virtual object features in the second image can be fused together to identify the virtual object from the second image.

[0112] The first image contains a detection box for the virtual object. It should be noted that during the virtual object recognition process, this detection box does not need to be displayed in the first image; the server only needs to record the detection box.

[0113] In some implementations, when initially recognizing virtual objects in image frames of the video to be recognized, it is necessary to manually mark the virtual objects in the first frame of the video where the virtual object first appears. This can be done by marking the virtual objects to be recognized in the first frame according to input annotation instructions, thus marking the position of the virtual objects in the first frame. For example, this can be done using detection boxes. The image frame where the virtual object first appears is then used as the first image, and the image frames adjacent to it are used as the second images for recognition. This process continues, allowing the detection boxes for virtual objects in each image frame of the video to be recognized starting from the first frame. Therefore, before step 102, the process further includes:

[0114] (102.a.1) Extract the first image frame containing the virtual object to be identified from the video to be identified;

[0115] (102.a.2) Label the virtual objects to be identified in the first image frame to obtain the target image frame containing the detection boxes of the labeled virtual objects.

[0116] Step 102, "Extracting the first and second images of adjacent frames from the video to be identified," may include:

[0117] (102.b.1) The target image frame is used as the previous frame image, and the previous frame image is determined as the first image;

[0118] (102.b.2) Based on the relationship between adjacent frames in the video frame to be identified, obtain the next video frame adjacent to the first image, and determine the next video frame as the second image.

[0119] Specifically, the first frame image in the video where the virtual object first appears is identified. This first frame image is not limited to the very first frame in the video to be identified; it can be understood as the initial image in which the virtual object first appears. A detection bounding box is used to label the virtual object in this first frame image as the first image. The next frame adjacent to the first frame image is selected as the second image. Furthermore, based on the display form of the virtual object in the first image at its first appearance, corresponding reference features are determined, providing a basis for identifying the virtual object in the second image. This facilitates the identification of the virtual object from the second image and the labeling of the identified virtual object.

[0120] For example, using a game video as the video to be identified, and the virtual object as the virtual player character in the game, in the first identification round, based on the identified virtual player character in the video, the initial game frame where the virtual player character first appears is determined, such as the frame containing the virtual player character at the beginning of the game video. A detection bounding box is used to mark the virtual player character in the initial game frame, resulting in a target image frame containing the detection bounding box for the virtual player character, which is used as the first image. Then, the next game frame adjacent to this initial game frame is obtained as the second image. It should be noted that the detection bounding box is not limited to rectangles, circles, finger shapes, scissor shapes, animal tracking markers, halos, etc.

[0121] Furthermore, after the virtual object in the second image is identified in the first recognition round, the second recognition round begins. Since the virtual object has already been identified in the "second image" of the first recognition round and marked with a detection box, in order to identify the virtual object from the next adjacent frame of the "second image," the "second image" of the first recognition round is used as the first image of the current second recognition round, and its next adjacent frame is used as the second image of the current second recognition round. This continues to identify virtual objects from the second image in the second recognition round, and the identified virtual objects are marked with detection boxes. The above process of acquiring the first and second images is repeated in a progressive manner until the virtual object no longer appears in a certain frame of the video to be identified, thus completing the identification of virtual objects in the entire video to be identified.

[0122] For example, using a video of a game match as the video to be identified, and the virtual object as the virtual player character in the game match, after identifying the virtual player character in the target game screen ("second image") in the previous identification round, the target game screen is used as the first image in the current identification round, and the next game screen adjacent to it is used as the second image to be identified in the current identification round, so as to identify and label the virtual player character from the second image in the current identification round.

[0123] It should be noted that when a virtual object in the video to be identified appears in stages, this "staged appearance" can be understood as the virtual object appearing intermittently in the video. In this case, the virtual object can be identified "stage by stage." For example, the virtual object appears continuously in frames 1 to 200 of the video to be identified and then disappears, and then appears in frames 800 to 1500 of the video to be identified. In the above situation, after identifying and labeling the virtual objects in the first 200 frames, the virtual objects can no longer be identified in the 201st frame. In order to continue to identify the virtual objects in the 800th to 1500th frames, the above-mentioned implementation method of "taking the image frame in which the virtual object first appears as the first image" can be followed. For example, the virtual object in the 800th frame can be labeled by the detection box and used as the first image, and the 802nd frame can be used as the second image. The same virtual object can be identified in the 800th to 1500th frames in this way until the 1501st frame can be used as the second image and the same virtual object can no longer be identified. Alternatively, when the 1500th frame is the last frame of the video to be identified, the identification can be stopped after the 1500th frame is identified.

[0124] By using the above method, the first and second images in each recognition round can be determined, so that the first image, which has already identified the virtual object, can be used as a reference for the second image, so that the virtual object can be identified and labeled from the second image to complete the recognition of the virtual object.

[0125] 103. Input the first image and the second image into the target model to obtain the target detection box of the virtual object in the second image.

[0126] In this embodiment of the application, in order to subsequently track and identify virtual objects in the video to be identified, the virtual objects in each image frame of the video to be identified can be identified by the target model, and the location area of ​​the identified virtual object can be marked so that the virtual object can be tracked and identified based on the target detection box in multiple consecutive frames of images.

[0127] To identify virtual objects in each image frame using a target model, the model needs to be trained. Specifically, the training process involves: acquiring a first sample image and a second sample image representing adjacent frames. Both images contain the same virtual object, and the first sample image includes an initial bounding box for the virtual object. The first and second sample images are used as input, along with the initial bounding box, as training labels to train the preset model. Training continues until the predicted bounding box in the second sample image output by the preset model matches the virtual object labeled in the initial bounding box, thus obtaining the trained target model.

[0128] The target model can enhance and fuse the region features of the virtual object within the detection box in the first image and the object color features in the second image, so as to predict the target detection box of the virtual object in the second image based on the fused aggregated feature matrix.

[0129] To better illustrate the embodiments of this application, the following concepts will be explained first:

[0130] Spatial perception mechanisms can sense the distribution of information features in an image, enabling the enhancement of corresponding features based on spatial perception to facilitate the extraction of target features. For example, spatial perception mechanisms can be used to perceive the feature distribution information of virtual objects in an image frame. This feature distribution could be the distribution of color features, generating spatial perception weights corresponding to the feature distribution. This, in turn, enhances the representation of the virtual object's features, ensuring accurate feature extraction of the virtual object subsequently.

[0131] Channel-aware mechanisms can sense the color channels corresponding to image features, allowing for weighted processing of the color features of virtual objects along the channel dimension. For example, by sensing the color channels of virtual object features within an image, channel-aware mechanisms can obtain corresponding channel weights, and then weight the virtual object features based on these weights. This makes the virtual object features more prominent relative to the background features, facilitating their differentiation from the background.

[0132] The image processing procedure for the target model is described by combining spatial perception and channel perception mechanisms, as follows:

[0133] First, when the target model labels the target detection box for the virtual object in the second image, it first enhances the representation of the relevant region features of the identified virtual object in the first image through a spatial perception mechanism, so as to facilitate the extraction of the region feature matrix of the identified virtual object from the first image; and second, it enhances the representation of the corresponding color features in the second image, so as to facilitate the extraction of the color feature matrix of the virtual object from the second image as much as possible.

[0134] Then, based on a channel-aware mechanism, the target model performs attention-weighted processing on the region feature matrix and the color feature matrix respectively, obtaining a first-channel feature matrix and a second-channel feature matrix. This allows the channel feature matrices to focus on enhancing the representation of relevant features of the virtual object, such as enhancing the color features of the virtual object, making the virtual object features more prominent relative to the background features. Furthermore, after weighting the feature matrices, convolution processing is performed on the first-channel and second-channel feature matrices respectively to further extract virtual object features, obtaining a target region feature matrix and a target color feature matrix. Further, the target region feature matrix and the target color feature matrix are fused according to the channel dimension, so that the fused feature matrix can simultaneously represent the enhanced spatial and color dimensions of the virtual object. Therefore, based on this fused feature matrix, the virtual object in the second image can be better identified, preventing the inability to recognize the virtual object due to its variability in the second image.

[0135] Finally, the target model determines the virtual objects in the second image based on the classification results of the fused aggregated feature matrix, and marks the location regions of the virtual objects in the second image using target detection boxes.

[0136] In some implementations, when the target model labels the location region of the virtual object in the second image, its feature processing mainly includes: a spatial awareness mechanism learning the location features of the virtual object, a channel awareness mechanism learning the channel features of the virtual object, and classification labeling. For example, step 103 may include:

[0137] (1) Input the first image and the second image into the target model;

[0138] (2) Extract the region features of the virtual object from the first image through the spatial attention mechanism to obtain the region feature matrix;

[0139] (3) Extract object color features from the second image using a spatial attention mechanism to obtain a color feature matrix;

[0140] (4) The region feature matrix and the color feature matrix are represented by channel attention enhancement and fused to obtain the aggregated feature matrix;

[0141] (5) Based on the aggregated feature matrix, the location regions of the virtual objects in the second image are labeled to obtain the target detection box.

[0142] Specifically, the framework of the target model mainly consists of three networks: a feature extraction network (A), a feature aggregation network (B), and a classification network (C). These three networks can be used to combine the features of virtual objects in the first image to identify virtual objects in the second image and to label the location regions of virtual objects in the second image.

[0143] First, the first and second images are input into the target model. The spatial awareness mechanism (i.e., spatial attention mechanism) in the feature extraction network (A) performs preliminary enhancement representations on the first and second images respectively, extracting the region features of the virtual object in the first image to obtain a region feature matrix, and extracting the color features of the object in the second image to obtain a color feature matrix. Then, the region feature matrix and the color feature matrix are fused by channel attention weighting through the feature aggregation network (B) to obtain an aggregated feature matrix. Finally, the classification network (C) labels the location of the virtual object in the second image based on the aggregated feature matrix, obtaining the target detection box. In this way, virtual objects can be accurately and quickly identified from the second image to be recognized, and the location regions of the virtual objects can be labeled, effectively avoiding the phenomenon that morphological differences between adjacent frames of the same virtual object affect recognition, thus improving the accuracy of virtual object recognition.

[0144] I. Description of the spatial perception mechanism for learning the location characteristics of virtual objects:

[0145] To enable the extraction of the region feature matrix of the virtual object from the first image and the color feature matrix from the second image, a spatial awareness mechanism can be used to enhance the representation of features in the first and second images, making the relevant regions in the images more prominent compared to the surrounding features. Furthermore, multiple convolutional layers are used to dynamically learn the relevant features of the first and second images to extract the region feature matrix of the virtual object from the first image and the object-related color feature matrix from the second image, thereby improving the reliability and accuracy of extracting the relevant features of the virtual object.

[0146] In some implementations, in order to extract the region feature matrix from the first image and the color feature matrix from the second image, the features in the first and second images can be enhanced based on the spatial attention corresponding to the spatial perception mechanism, so as to enhance the features of the virtual objects in the images and facilitate the feature extraction of the virtual objects. For example, taking the extraction of the color feature matrix corresponding to the second image as an example, step (3) may include:

[0147] (3.1) Perform convolution processing on the second image to obtain the feature matrix of the second image;

[0148] (3.2) Based on the spatial attention mechanism, the color feature weights are determined from the second image feature matrix, and the object color features in the second image feature matrix are enhanced by attention according to the color feature weights to obtain the color feature matrix.

[0149] Specifically, the spatial awareness mechanism learns the positional features of the virtual object to generate corresponding attention weights, which are used to enhance the representation of the virtual object's features in the second image. This allows for the extraction of the virtual object's color feature matrix from the second image. This process can be implemented through the feature extraction network (A) in the target model. This implementation first introduces the structure of the feature extraction network (A), as follows:

[0150] (A) Introduction to Feature Extraction Networks:

[0151] This system is primarily used to extract features from a first image and a second image, respectively. It enhances the feature representation of virtual objects in the second image by using spatially perceptual weights in two separate steps, thereby extracting a feature matrix related to the virtual objects. The feature extraction network comprises two sub-networks with identical structures, arranged in parallel. These sub-networks are used to perform feature spatial perception on the first and second images, respectively, to enhance the representation of virtual objects in the images and complete feature extraction.

[0152] For ease of understanding, the two feature extraction sub-networks can be defined as (A.1) and (A.2) respectively. Each feature extraction sub-network contains at least three types of modules: convolutional layers, spatial awareness modules, and feature fusion modules.

[0153] For example, for the first feature extraction sub-network (A.1), the feature processing procedure is as follows:

[0154] First, the first image is convolved using a convolutional layer to obtain a first image feature matrix. Then, the spatial awareness module, with its corresponding spatial attention mechanism, determines the first spatial region features of the virtual object's location from the first image feature matrix; this can be understood as region feature weights. Next, the feature fusion module fuses these region feature weights into the first image feature matrix, enhancing the representation of the region features according to these weights, thus obtaining a first fused feature matrix. Finally, the first fused feature matrix is ​​convolved using a convolutional layer to obtain the region feature matrix corresponding to the first image.

[0155] For example, the feature processing procedure for the second feature extraction sub-network (A.2) is as follows:

[0156] First, the second image is convolved using a convolutional layer to obtain the second image feature matrix. Then, a spatial awareness module with a corresponding spatial attention mechanism determines the second spatial weights of the object's color features from the second image feature matrix; these weights can be understood as color feature weights. Next, a feature fusion module fuses these color feature weights into the second image feature matrix, performing attention-enhanced representation of the color features according to their weights, which can be considered as obtaining the second fused feature matrix. Finally, the second fused feature matrix is ​​convolved using a convolutional layer to obtain the color feature matrix corresponding to the second image.

[0157] In some implementations, spatial perception weights can be obtained twice, using a spatial perception mechanism (spatial attention mechanism), and then fused with the image feature matrix to further enhance the representation of the virtual object's relevant features. Step (3.2) may then include:

[0158] (3.2.1) Based on the spatial attention mechanism, the initial color feature weights are determined from the color features in the second image feature matrix;

[0159] (3.2.2) Based on the initial color feature weights, the object color features in the second image feature matrix are initially enhanced to obtain the initial fusion feature matrix;

[0160] (3.2.3) Perform convolution on the initial fused feature matrix to obtain the initial color feature matrix;

[0161] (3.2.4) The color features in the initial color feature matrix are perceptually processed by the spatial attention mechanism, and the object color features in the initial fusion feature matrix are enhanced according to the target color feature weights obtained by the perceptual processing, so as to obtain the target fusion feature matrix.

[0162] Specifically, spatial awareness weights are generated through the attention mechanism of two spatial awareness modules, and the features of the virtual object's position are dynamically learned in multiple convolutional layers to enhance the representation of the virtual object features in the second image feature matrix. It should be noted that this process may include two rounds of acquiring the spatial awareness weights.

[0163] For ease of understanding, this embodiment will describe step (3.2) in detail with respect to each module within the feature extraction subnetwork. It should be noted that the processing procedures of the first feature extraction subnetwork (A.1) and the second feature extraction subnetwork (A.2) are identical when enhancing the feature representation of the virtual object using spatially aware weights. Therefore, in this embodiment, only the processing flow of one feature extraction subnetwork is described; the processing flow of the other feature extraction subnetwork can be referred to the same described processing flow.

[0164] For example, taking (A.1) as an example, the second feature extraction sub-network contains at least 3 convolutional layers, 2 spatial awareness modules, and 2 feature fusion modules. Specifically, according to the feature processing order of each module, the 3 convolutional layers are defined as the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively; the 2 spatial awareness modules are defined as the first spatial awareness module and the second spatial awareness module, respectively; and the 2 feature fusion modules are defined as the first feature fusion module and the second feature fusion module, respectively. For ease of understanding, step (3.2) is described in conjunction with (A.2). The processing procedure for the second image within (A.2) is as follows:

[0165] First, the second image is convolved through the first convolutional layer inside (A.2) to extract image features, resulting in the second image feature matrix. Then, the first step of obtaining spatially aware weights can begin.

[0166] Specifically, in the first round of acquiring spatial awareness weights, the processing procedures of other modules may include:

[0167] (A.2.1) The initial color feature weights are determined from the color features in the second image feature matrix through the spatial attention mechanism in the first spatial perception module.

[0168] (A.2.2) The initial color feature weights are fused into the second image feature matrix through the first feature fusion module, thereby achieving a preliminary enhanced representation of the object color features in the second image feature matrix and obtaining the initial fused feature matrix.

[0169] (A.2.3) The initial fused feature matrix is ​​convolved through the second convolutional layer to extract the color features of the virtual object from the initially enhanced feature matrix, thus obtaining the initial color feature matrix.

[0170] (A.2.4) The spatial attention mechanism in the second spatial perception module is used to perform perceptual processing on the color features in the initial color feature matrix in order to determine the target color feature weights from the color features in the initial color feature matrix.

[0171] (A.2.5) The target color feature weights are fused with the initial color feature matrix through the second feature fusion module, thereby enhancing the representation of the object color features in the initial color feature matrix and obtaining the target fused feature matrix.

[0172] (A.2.6) The target fusion feature matrix is ​​convolved by the third convolutional layer to obtain the color feature matrix.

[0173] In this way, spatial weights are obtained sequentially through the spatial attention mechanism in the first spatial perception module and the second spatial perception module, and the features of the virtual object's position are dynamically learned in multiple convolutional layers to enhance the color features of the virtual object in the second image feature matrix. Thus, the color feature matrix of the virtual object is extracted through the final convolutional processing.

[0174] To facilitate understanding, the following description of the spatial weight acquisition process, in conjunction with the various units within the spatial perception module, will illustrate the learning process of the spatial perception mechanism. It should be noted that the internal structures of the first and second spatial perception modules can be identical, used to acquire spatial weights sequentially, and the processing flow for acquiring spatial weights is also consistent for both. In this embodiment, the first spatial perception module will be used as an example, and step (3.2.1) will be described in conjunction with the various sub-units within the first spatial perception module to illustrate the process of acquiring the initial color feature weights.

[0175] In some implementations, the initial color feature weights can be determined from the second image feature matrix based on the spatial attention mechanism corresponding to the first spatial perception module. Step (3.2.1) may specifically include:

[0176] (3.2.1.1) Based on the spatial attention mechanism, the second image feature matrix is ​​convolved to obtain the first convolution feature matrix;

[0177] (3.2.1.2) Perform eigenvalue normalization on the first convolutional feature matrix, and perform local response normalization on the eigenvalues ​​in the initial eigenvalue matrix obtained by normalization to obtain the target eigenvalue matrix;

[0178] (3.2.1.3) Perform convolution on the target feature matrix to obtain the second convolution feature matrix;

[0179] (3.2.1.4) Perform color weight classification on the color features in the second convolution feature matrix to obtain the initial color feature weights.

[0180] Specifically, the first spatial perception module includes at least a first convolutional sub-unit, a rectified linear unit (ReLU), a local response normalization unit (LRN), a second convolutional sub-unit, and a classification unit (sigmoid function). The specific process by which the first spatial perception module obtains the initial color feature weights is as follows:

[0181] First, the second image feature matrix is ​​convolved by the first convolution sub-unit to extract the color features of the virtual object from the second image feature matrix, thus obtaining the first convolution feature matrix.

[0182] Then, the eigenvalues ​​in the first convolutional feature matrix are normalized using a rectified linear unit (CLU), so that all eigenvalues ​​less than 0 are assigned the value 0, while the original eigenvalues ​​greater than 0 are preserved. The output of the CLU is determined in this way, resulting in the normalized initial eigenvalue matrix. It can be understood that any eigenvalue in this initial eigenvalue matrix is ​​greater than or equal to 0.

[0183] Furthermore, the initial eigenvalue matrix is ​​localized using a local response normalization unit. This unit suppresses neurons with small feedback, aiming to suppress some eigenvalues ​​in the initial eigenvalue matrix to enhance the weighting coefficients of the color features of the virtual object. Specifically, in this embodiment, for any eigenvalue in the initial eigenvalue matrix, when a relatively small eigenvalue is input to a neuron in the local response normalization unit, the feedback is small, and the output of that neuron is suppressed. Conversely, when a relatively large eigenvalue is input to a neuron in the local response normalization unit, the response is large, and the output of that neuron is not suppressed. Normalization can then be performed under this suppression, mapping any eigenvalue in the initial eigenvalue matrix that was originally greater than or equal to 0 to between 0 and 1, resulting in the target eigenvalue matrix, which is then used to enhance the spatial weighting coefficients of the virtual object.

[0184] For example, the eigenvalues ​​in the initial eigenvalue matrix include 0, 1, 2, 6, and 8, where eigenvalues ​​0, 1, and 2 represent background eigenvalues, and 6 and 8 typically represent eigenvalues ​​of the virtual object. Clearly, eigenvalue 8 shows the best response in the local response normalization unit, and eigenvalue 6 also performs well. However, eigenvalues ​​0, 1, and 2 have relatively smaller responses in the local response normalization unit. Therefore, after local response normalization, the corresponding eigenvalues ​​can be 0, 0.001, 0.009, 0.8, and 0.99, respectively. Thus, by normalizing the initial eigenvalue matrix using the local response normalization unit, the weights of the background features in the source image are suppressed, making the color feature weights at the virtual object's location greater than the background feature weights, thereby optimizing the spatial weight coefficients of the virtual object's color features.

[0185] Next, the target feature value matrix is ​​convolved by the second convolution sub-unit to extract features and obtain the second convolution feature matrix, so that the spatial weight coefficient of the virtual object features can be improved based on the optimized feature values.

[0186] Finally, the sigmoid function is used as the classification unit to generate initial color feature weights based on the second convolutional feature matrix. Specifically, the sigmoid function is used to non-linearly assign values ​​to each feature value in the second convolutional feature matrix, ensuring that the assigned feature values ​​are between 0 and 1, thus obtaining spatial weights, i.e., initial color feature weights. This further optimizes the weight coefficients for enhancing the virtual object features, allowing for the initial enhancement of the virtual object features in the second image feature matrix based on these initial color feature weights, making the virtual object features more prominent relative to the background features.

[0187] Following the above method, the target color feature weights can be obtained from the initial color feature matrix through the second spatial perception module.

[0188] II. Description of the channel-aware mechanism for learning the channel characteristics of virtual objects:

[0189] To enable the subsequent fusion of the region feature matrix of the virtual object in the first image with the color feature matrix in the second image, a channel-aware mechanism (channel attention mechanism) can be used to enhance the feature values ​​of the region feature matrix and the color feature matrix. This strengthens the relevant features of the virtual object, increases the feature value difference between the object features and the background features, and makes the relevant features of the virtual object more prominent relative to the surrounding background features. Then, convolutional layers are used to separate the relevant features of the virtual object in the region feature matrix and the color feature matrix, and fuse them on the channel to obtain an aggregated feature matrix.

[0190] In some implementations, after feature extraction is performed on the first image and the second image respectively, the feature aggregation stage can be entered. This stage mainly involves weighting and fusing the distinguishing feature matrix and the color feature matrix. Specifically, step (4) may include:

[0191] (4.a.1) The region feature matrix is ​​perceptually processed based on the channel attention mechanism to obtain the first channel weight, and the region feature matrix is ​​represented by channel attention enhancement based on the first channel weight to obtain the first channel feature matrix;

[0192] (4.a.2) Perform perceptual processing on the color feature matrix based on the channel attention mechanism to obtain the second channel weights, and perform channel attention enhancement representation on the color feature matrix based on the second channel weights to obtain the second channel feature matrix;

[0193] (4.a.3) Perform convolution processing on the first channel feature matrix and the second channel feature matrix respectively to obtain the target region feature matrix and the target color feature matrix;

[0194] (4.a.4) The target region feature matrix and the target color feature matrix are fused on the channel to obtain the aggregated feature matrix.

[0195] Specifically, in the feature aggregation stage, corresponding channel attention weights are learned from the region feature matrix and color feature matrix through a channel-aware mechanism, resulting in first channel weights and second channel weights, which are then used to weight the region feature matrix and color feature matrix, respectively. Furthermore, the weighted feature matrices are convolved and fused to obtain an aggregated feature matrix, which is used for subsequent recognition of virtual objects in the second image. This feature aggregation stage can be implemented using a feature aggregation network (B) in the target model. This implementation first introduces the structure of the feature aggregation network (B), as follows:

[0196] (B) Introduction to the Feature Aggregation Network: The feature aggregation network determines the channel-aware weights corresponding to the region feature matrix and color feature matrix through each module, and then fuses them after weighted processing. The feature aggregation network may include a channel-aware module (B.1), a first weighting module (B.2), a second weighting module (B.3), a first convolutional module (B.4), a second convolutional module (B.5), and a feature fusion module (B.6). The feature aggregation process based on these modules is as follows:

[0197] First, based on the channel attention mechanism corresponding to the channel perception module (B.1), the region feature matrix is ​​processed to obtain the first channel weight; then, the first weighting module (B.2) is used to perform channel attention enhancement representation on the region feature matrix according to the first channel weight to obtain the first channel feature matrix.

[0198] Then, based on the channel attention mechanism corresponding to the channel perception module (B.1), the color feature matrix is ​​processed to obtain the second channel weights; and the second weighting module (B.3) is used to perform channel attention enhancement representation on the color feature matrix according to the second channel weights to obtain the second channel feature matrix.

[0199] Next, the first channel feature matrix is ​​convolved by the first convolution module (B.4) to obtain the target region feature matrix; and the second channel feature matrix is ​​convolved by the second convolution module (B.5) to obtain the target color feature matrix.

[0200] Finally, the feature fusion module (B.6) fuses the target region feature matrix and the target color feature matrix in the channel to obtain the aggregated feature matrix.

[0201] Through the above process, the region feature matrix and color feature matrix are weighted and fused in the channels to reconstruct the feature description (aggregated feature matrix) for virtual objects. The aggregated feature matrix can then be used in the subsequent classification process, effectively avoiding the phenomenon that morphological differences between adjacent frames of the same virtual object affect subsequent recognition, thus enabling the recognition of virtual objects.

[0202] (B.1) Regarding the process of the channel awareness module acquiring channel weights:

[0203] In the implementation of this application, when determining the first channel weight and the second channel weight through the channel sensing mechanism of the channel sensing module, the sensing process of the first channel weight and the second channel weight can be executed separately, or the channel sensing weight process can be executed simultaneously.

[0204] (B.1.a) Perform the perception process of the first channel weight and the second channel weight separately:

[0205] In some implementations, the process of determining channel weights using a channel attention mechanism may include fully connected processing, removal of non-compliant feature values, and weight integration. For example, taking the perception process of the second channel weights as an example, step (4.2) of "performing perceptual processing on the color feature matrix based on the channel attention mechanism to obtain the second channel weights" may include:

[0206] (4.2.1) Based on the channel attention mechanism, the color feature matrix is ​​processed by full connection to obtain the first feature matrix;

[0207] (4.2.2) Based on feature compliance, some feature values ​​in the first feature matrix are hidden to obtain the second feature matrix;

[0208] (4.2.3) The second feature matrix is ​​integrated with color weights by activation function to obtain the second channel weights.

[0209] Specifically, when the channel-aware mechanism learns the weights of the second channel from the color feature matrix, it can be achieved through various units in the channel-aware module. This channel-aware module may include fully connected layers, Rectified Linear Units (ReLU), feature scaling units (Dropout), and weight integration layers (softmax function + sigmoid function), etc. For ease of understanding, this embodiment will describe the process of perceiving the weights of the second channel in conjunction with the various units within the channel-aware module. (B.1.a) The specific process is as follows:

[0210] (B.1.a.1) Based on the fully connected layer in the channel sensing module, the color feature matrix is ​​processed by full connection. The fully connected layer may include a first fully connected layer and a second fully connected layer. The first fully connected layer connects each color feature in the color feature matrix to obtain an initial connected feature matrix. Then, the second fully connected layer connects and classifies each feature in the initial connected feature matrix to obtain a first feature matrix.

[0211] It should be noted that during the fully connected processing, the eigenvalues ​​in the feature matrix can also be normalized using a rectified linear unit (ReLU). For example, after classifying the features in the initial connected feature matrix through a second fully connected layer, the eigenvalues ​​in the initial connected feature matrix can be normalized using a rectified linear unit, so that all eigenvalues ​​less than 0 are output as 0, and eigenvalues ​​greater than 0 are directly output, resulting in a first feature matrix with eigenvalues ​​greater than 0.

[0212] (B.1.a.2) A feature scaling unit (Dropout) is used to hide some feature values ​​in the first feature matrix. Specifically, the feature scaling unit has neuron scaling properties, which can be used to partially scale neurons. That is, neurons are scaled according to a certain scaling factor (such as scaling probability, scaling percentage of neurons, or scaling number) to make neurons inactive, thus hiding the representation of some neurons. As a result, the corresponding feature values ​​after hiding some feature values ​​are 0, resulting in a second feature matrix with the hidden feature values. Thus, when hiding some features in the first feature matrix using the feature scaling unit, dirty data in the first feature matrix can be hidden. Specifically, non-compliant features output by the fully connected layer can be hidden. For example, non-compliant features can be understood as image sub-features that do not conform to the relevant entities in the second image, such as the color features of disordered or confused virtual objects. By hiding the non-compliant feature values, a compliant second feature matrix is ​​obtained.

[0213] (B.1.a.3) According to the dimension of color channels, the softmax function in the weight integration layer calculates the probability of each color channel feature in the second feature matrix, so that the color represented by each feature value is represented by a probability value. For example, the sum of the probability values ​​of all colors is 1. For example, R, G, and B are 0.2, 0.3, and 0.5 respectively, and the sum is 1, thus obtaining the probability corresponding to each color. Then, the softmax function outputs the probability distribution of the feature of each color channel, and uses the probability distribution as the score value of the feature of each color channel to obtain the preliminary channel weight matrix.

[0214] Furthermore, the initial channel weight matrix is ​​activated using the sigmoid function, and each score value in the initial channel weight matrix is ​​non-linearly assigned a value so that the weight coefficient of each color channel is between 0 and 1. For example, the feature of the R color channel (probability value of 0.2) corresponds to a weight of 0.52, the feature of the G color channel (probability value of 0.3) corresponds to a weight of 0.535, and the feature of the B color channel (0.5) corresponds to a weight of 0.573. In this way, the color channel weights are obtained by integration.

[0215] In this embodiment of the application, since the sensing process of the first channel weight and the second channel weight is consistent, the sensing process of the first channel weight can be executed separately according to the above process (B.1.a.1)-(B.1.a.3).

[0216] (B.1.b) Regarding the perception process of simultaneously executing the first channel weight and the second channel weight:

[0217] In this embodiment of the application, since the feature aggregation network needs to process a very large amount of feature data when performing channel perception, in order to reduce the amount of data computation and improve the performance of the model when recognizing virtual objects, the perception process of the first channel weight and the second channel weight can be executed simultaneously.

[0218] In some implementations, to achieve simultaneous perception of the first and second channel weights, it is necessary to concatenate the region feature matrix and color feature matrix output by the feature extraction network, and then apply channel weights to the concatenated feature matrix using the channel attention mechanism corresponding to the perception module. For example, step (4) may include:

[0219] (4.b,1) Concatenate the region feature matrix and the color feature matrix to obtain the connection feature matrix;

[0220] (4.b.2) Perform a full connection process on the connection feature matrix to obtain the region connection feature matrix and the color connection feature matrix;

[0221] (4.b.3) Determine the first channel weight from the region connectivity feature matrix based on the channel attention mechanism;

[0222] (4.b.4) Based on the channel attention mechanism, the weight of the second channel is determined from the color connection feature matrix.

[0223] It should be noted that, in order to achieve simultaneous perception of the first and second channel weights, the feature aggregation network may also include a feature connection module for connecting the region feature matrix and the color feature matrix in step (4.b.1). This connection process can be conat, which is only a connection operation and not a feature fusion process. The implementation process of steps (4.b.2), (4.b.3), and (4.b.4) is equivalent to the perception process of the second channel weight in (B.1.a.1)-(B.1.a.3), and will not be elaborated further here.

[0224] Furthermore, after determining the weights of the first and second channels, the virtual object features in the region feature matrix and color feature matrix can be weighted respectively to increase the difference in feature values ​​between the object features and the background features. Then, after two processes of convolution and fusion, the fused aggregated feature matrix can be obtained.

[0225] III. Introduction to Classification Networks in the Virtual Object Recognition Process:

[0226] After fusing to obtain the aggregated feature matrix, this application can identify virtual objects based on the aggregated feature matrix and mark the location regions of the identified virtual objects in the second image to generate target detection boxes.

[0227] In some implementations, the aggregated feature matrix can be fully connected, and the target feature matrix obtained from the fully connected processing can be classified to identify and label virtual objects in the second image based on the classification results. For example, step (5) may include:

[0228] (5.a.1) Perform a fully connected operation on the aggregated feature matrix to obtain the target feature matrix;

[0229] (5.a.2) Perform binary classification on the target feature matrix to obtain the object modality value;

[0230] (5.a.3) Based on the object modality value, the location region of the corresponding virtual object in the second image is labeled to obtain the target detection box.

[0231] Specifically, a classification network in the target model is used to identify virtual objects in the second image, and the location regions of the virtual objects in the second image are labeled. This classification network (C) may include a fully connected layer (C.1) and a binary classification module (C.2). The process of identifying virtual objects using the fully connected layer and the binary classification module is described as follows:

[0232] First, the aggregated feature matrix is ​​processed through a fully connected layer in the classification network. This fully connected layer includes a first connected layer and a second connected layer. The first connected layer connects the features in the aggregated feature matrix to obtain a connected feature matrix. Then, the connected feature matrix undergoes connection classification processing to obtain the target feature matrix. Next, the binary classification module in the classification network calculates modal feature values ​​for the target feature matrix to obtain object modal values, which represent the corresponding virtual objects. Finally, the corresponding objects are determined based on the object modal values, and the locations of the virtual objects in the second image are labeled to obtain the corresponding target detection boxes.

[0233] It should be noted that the binary classification module has a corresponding virtual object template during the training phase. This virtual object template can be understood as an object classification threshold or object classification interval, which is only used for the recognition of a specific virtual object. Therefore, after calculating the object modality value, the binary classification module can compare the object modality value with the corresponding object classification threshold or object classification interval to determine the virtual object corresponding to the object modality value. Then, the location region of the virtual object identified in the second image is labeled using the target detection box.

[0234] In some implementations, object modality values ​​can be calculated by combining relevant classification weights and constant experience. For example, step (5.a.2) may include:

[0235] (5.a.2.1) Obtain the feature hiding parameters of the target model in the channel attention-enhanced representation, and calculate the feature classification weights based on the feature hiding parameters;

[0236] (5.a.2.2) The target feature matrix is ​​weighted according to the feature classification weights to obtain the target weight feature matrix;

[0237] (5.a.2.3) The matrix difference between the target weight feature matrix and the constant bias matrix is ​​accumulated to obtain the object modal value.

[0238] Specifically, to enhance the performance of the binary classification module, object modality values ​​can be calculated by combining feature classification weights and constant factors. The feature classification weights can be determined based on the feature hiding parameters during the channel perception stage. For example, they can be determined by the scaling of neurons in the Dropout stage of channel perception. The scaling of these neurons reflects the parameters used during feature hiding; for instance, the scaling percentage of neurons is equal to the proportion (number) of hidden features. Therefore, the feature classification weights can be determined based on the scaling percentage of neurons. The constant factor can be preset based on empirical values ​​and is not limited here. In this way, by combining feature classification weights and constant factors to calculate object modality values, the classification effect can be enhanced during binary classification, enabling the identification of corresponding virtual objects.

[0239] In some implementations, when the first image contains multiple different virtual objects, it is necessary to identify multiple virtual objects from the second image and mark the location region of each virtual object in the second image. It should be noted that when the first image and the second image frame contain multiple different virtual objects, after spatial perception and feature extraction in the feature extraction stage, the resulting region feature matrix and color feature matrix respectively contain the relevant features of multiple virtual objects; furthermore, in the feature aggregation stage, the fused aggregated feature matrix contains the features of multiple virtual objects. Then step (5) may include:

[0240] (5.1) Perform a fully connected operation on the aggregated feature matrix to obtain a target feature matrix containing features of multiple virtual objects;

[0241] (5.2) The target feature matrix is ​​subjected to binary classification by different binary classification modules in the target model to obtain the object modality value corresponding to each virtual object feature;

[0242] (5.3) Based on the object modality value, the location region of each virtual object in the second image is labeled to obtain the target detection box corresponding to each virtual object.

[0243] It should be noted that when identifying multiple virtual objects in an image frame using a target model, multiple binary classification modules can be set up in the binary classification network. Each binary classification module is used to identify a specific virtual object. Thus, multiple binary classification modules calculate the object modality value based on the features of each virtual object in the target feature matrix. Specifically, after performing binary classification, each binary classification module can determine whether it belongs to the modality of the virtual object to be identified based on the comparison result between the object modality value and a threshold. Virtual objects belonging to the current modality are identified as positive samples, and the virtual objects corresponding to the positive samples in the second image are labeled using target detection boxes.

[0244] Using the above method, virtual objects in each image frame of the video to be identified can be identified through the target model. The identification process involves enhancing the representation of the features of the virtual objects in the image frame to accurately extract the features of the virtual objects, and weighting the extracted features to make the features of the virtual objects more prominent than the background features. Then, through convolution and fusion, the virtual objects are identified based on the aggregated features, and the location regions of the identified virtual objects are marked so that the virtual objects can be tracked and identified in subsequent target detection boxes in multiple consecutive frames of images.

[0245] 104. Based on the target detection box in the second image, identify the virtual objects in the video to be identified.

[0246] In this embodiment, after identifying and labeling the virtual objects in each video frame of the video to be identified, the virtual objects in the video to be identified can be tracked and identified based on the target detection box in the second image. It should be noted that the "second image" refers to each frame of the image to be identified in the process of the aforementioned target model identifying virtual objects, that is, it refers to any "second image" in multiple recognition rounds.

[0247] The shape of the target detection box is not limited to fingers, arrows, rectangles, circles, etc., and is not restricted here.

[0248] In some implementations, when tracking and identifying a virtual object in motion within the video to be identified, it can be understood as identifying the virtual object based on its motion trajectory. For example, step (104) may include: generating a target motion trajectory corresponding to the virtual object based on the target detection box in the second image; and identifying the virtual object based on the motion trajectory.

[0249] It should be noted that when generating motion trajectories, it is necessary to combine the target detection boxes in the second image with the temporal relationship of the video frames. Specifically, the "second image" can be sorted according to the temporal relationship of the video frames to obtain a graphic sequence containing target detection boxes. Then, according to the target detection boxes in the image sequence, the corresponding motion trajectory is constructed so as to track and identify virtual objects based on the motion trajectory.

[0250] For example, taking a game match scenario as an example, the game match video contains virtual player characters. After identifying and labeling the virtual objects in each game frame through the target model, the motion trajectory of the virtual objects can be generated based on the target detection boxes in each labeled game frame, according to the video frame coherence between multiple game frames. This motion trajectory is then used to track and identify the virtual player characters, so as to score the virtual player characters' gameplay.

[0251] For example, taking a movie video as an example, the movie video contains characters. After identifying and labeling the virtual objects in each movie image frame through the target model, the motion trajectory of the characters can be generated based on the target detection box in each labeled movie image frame according to the video frame coherence between multiple image frames. In order to track and identify the characters according to the motion trajectory, so as to score the characters' acting skills, appearance frequency, movement trajectory, etc.

[0252] Using the above methods, virtual objects in a video can be tracked and identified based on multiple labeled target detection boxes, ensuring the reliability of the tracking and identification process.

[0253] As can be seen from the above, the embodiments of this application can acquire a video to be identified and determine the virtual object to be identified in the video; extract a first image and a second image from adjacent frames of the video to be identified, wherein the first image contains the detection box of the virtual object; input the first image and the second image into a target model to obtain the target detection box of the virtual object in the second image; wherein the target model enhances and fuses the region features of the virtual object in the detection box in the first image and the object color features of the second image to predict the target detection box of the virtual object in the second image based on the fused aggregated feature matrix; and identify the virtual object in the video to be identified based on the target detection box in the second image. Therefore, this scheme can identify the virtual object to be tracked and identified in the video to be identified, and extract a first image to determine the location of the virtual object and a second image to be identified based on the relationship between adjacent frames from the video to be identified. The first and second images are then enhanced with feature representations in the spatial and channel dimensions using a target model to highlight the shape of the virtual object in the image, making the virtual object more prominent relative to other background elements. Furthermore, the virtual object features extracted from the first and second images are fused, and the virtual object to be identified in the second image is determined based on the aggregated feature matrix. Target detection boxes are then generated to identify the virtual object based on the motion trajectory corresponding to a series of target detection boxes. In this way, the same virtual object can be identified and labeled sequentially from each image frame, effectively avoiding the influence of the variability of virtual objects in the image on the recognition process, and improving the efficiency and accuracy of virtual object recognition.

[0254] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0255] This application takes virtual object recognition as an example to further describe the virtual object recognition method provided in this application.

[0256] Figure 3 This is a schematic diagram of another step in the virtual object recognition method provided in the embodiments of this application; Figure 4This is a schematic diagram of the architecture of the virtual object recognition system provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the steps of the virtual player character recognition method in game videos provided in this application embodiment; Figure 6 This is a schematic diagram of the interface of a game screen including a target detection box, provided in an embodiment of this application; Figure 7 This is a schematic diagram of the target model structure in the virtual object recognition method provided in this application embodiment; Figure 8 This is a schematic diagram of the spatial perception module provided in an embodiment of this application; Figure 9 This is a schematic diagram of the channel sensing module provided in an embodiment of this application. For ease of understanding, this embodiment of the application combines... Figure 3-9 Describe it.

[0257] In this embodiment, the description will focus on a virtual object recognition device, which can be integrated into computer devices such as terminals and servers. A communication connection exists between the terminal and the server to transmit data for processing. When the processors on the terminal and server execute the program instructions corresponding to the virtual object recognition method, the specific flow of the virtual object recognition method is as follows:

[0258] 201. The terminal sends the video to be recognized and the object recognition task to the server.

[0259] It should be noted that the terminal can be a terminal used by relevant personnel to select the video to be identified and to establish an object recognition task. This object recognition task can be a task targeting virtual characters, but it is not limited to including object information of the virtual objects to be identified.

[0260] For example, taking game videos as an example, game backend administrators can select the video to be identified and determine the virtual player characters that need to be identified in the video. They can then create an object recognition task based on the object information of the virtual player characters to be identified, such as their identifiers, names, and object attributes (e.g., appearance attributes). Finally, the game video and the object recognition task are sent to the server.

[0261] For example, taking a movie video as an example, in order to rate the characters in the movie video, the characters can first be marked and tracked for identification. Therefore, an object recognition task can be established based on object information such as the character's clothing, name, and actor information. Then, the movie video and the object recognition task are sent to the server.

[0262] 202. The server determines the virtual objects to be identified in the video based on the object recognition task.

[0263] After receiving the object recognition task and the video to be recognized, the server can determine the virtual object to be recognized in the video based on the object information in the object recognition task.

[0264] 203. The server extracts the first image frame containing the virtual object to be identified, and annotates the virtual object to be identified in the first image frame to obtain the target image frame containing the detection box of the annotated virtual object.

[0265] To enable frame-by-frame recognition of virtual objects in multiple image frames of a video, after identifying the virtual objects to be recognized, the first frame in the video where the virtual object first appears can be marked. This mark is then used to annotate the virtual object in the first frame using a target detection bounding box. This completes the recognition of the virtual object in the first frame, allowing subsequent image frames containing the annotations to be used as the basis for virtual object recognition in the next frame, thus achieving frame-by-frame recognition of multiple subsequent image frames.

[0266] 204. The server uses the target image frame as the previous image frame.

[0267] The server determines the current recognition progress of virtual objects in the video to be recognized and uses the image frame with the detection box marked as the previous frame.

[0268] 205. The server determines the previous frame image as the first image, and obtains the second image adjacent to the first image based on the relationship between adjacent frames in the video frame to be identified.

[0269] It should be noted that the tracking and recognition of virtual objects is primarily applied in video or image stream scenarios. By sequentially identifying and labeling virtual objects in each image or frame, the same virtual object can be tracked and identified throughout the entire video or image stream. Therefore, this recognition process is a gradual one, consisting of multiple recognition rounds. For example, identifying a virtual object in an image frame and marking it with a detection box is considered one recognition round.

[0270] For each recognition round, the previous frame image that has been recognized is taken as the first image, and the next adjacent frame image of the first image is taken as the second image.

[0271] 206. The server inputs the first and second images into the target model.

[0272] In order to enable subsequent tracking and recognition of virtual objects in the video to be recognized, the server can input the first image and the second image into the target model. The target model can then identify the virtual objects in each image frame of the video to be recognized and mark the location of the identified virtual objects so that the virtual objects can be tracked and recognized in subsequent images based on the target detection boxes in multiple consecutive frames.

[0273] Specifically, the framework of this target model mainly consists of three networks: a feature extraction network, a feature aggregation network, and a classification network.

[0274] The feature extraction network has convolution and spatial awareness mechanisms, which can enhance the feature representation of the first and second images to effectively and selectively extract the relevant features of the virtual object.

[0275] Among them, the feature aggregation network has a channel awareness mechanism and feature fusion, which is used to extract channel attention weights to weight the relevant features of virtual objects, making the features of virtual objects more prominent than background features, and then fusion after further feature extraction.

[0276] Among them, the classification network has the function of classifying and recognizing virtual objects, so as to identify virtual objects and label them through target detection boxes.

[0277] 207. The server extracts the regional features of the virtual object from the first image through a spatial attention mechanism to obtain a regional feature matrix, and extracts the color features of the object from the second image to obtain a color feature matrix.

[0278] Specifically, feature extraction can be achieved based on a feature extraction network. This feature extraction process includes: enhancing the features of the location of virtual objects in the image based on a spatial attention mechanism, and then extracting the features.

[0279] Specifically, based on the spatial awareness mechanism in the feature extraction network, the representation of features in the first and second images is enhanced so that the relevant regions in the images are more prominent than the surrounding features. Furthermore, the relevant features of the first and second images are dynamically learned through multiple convolutional layers to extract the region feature matrix of the virtual object from the first image and the object-related color feature matrix from the second image, thereby improving the reliability of extracting the relevant features of the virtual object.

[0280] It should be noted that spatial perception weights can be obtained by using the spatial perception mechanism (spatial attention mechanism) twice, and then fused with the image feature matrix to enhance the representation of relevant features of virtual objects, thereby facilitating the extraction of virtual object features.

[0281] 208. The server performs channel attention enhancement on the region feature matrix and color feature matrix and then fuses them to obtain the aggregated feature matrix.

[0282] Feature aggregation is achieved through a feature aggregation network.

[0283] Specifically, the feature values ​​of the region feature matrix and the color feature matrix are enhanced by the channel awareness mechanism (channel attention mechanism) in the feature aggregation network, thereby increasing the feature value difference between the object features and the background features, so that the relevant features of the virtual object are more prominent relative to the surrounding background features; then, the relevant features of the virtual object in the region feature matrix and the color feature matrix are separated by convolutional layers and fused on the channel to obtain the aggregated feature matrix.

[0284] 209. The server annotates the location regions of virtual objects in the second image based on the aggregated feature matrix, thus obtaining the target detection box.

[0285] Specifically, the classification network in the target model is used to identify virtual objects in the second image, and the location regions of the virtual objects in the second image are labeled.

[0286] Specifically, the aggregated feature matrix is ​​processed through a fully connected layer in the classification network to connect the features within the matrix. The features of virtual objects already identified in the previous frame are used as the basis for identifying the virtual objects to be identified in the current frame. This allows for the classification of virtual objects in the current frame by combining the already identified features, effectively avoiding the impact of variable or partially occluded virtual objects in the current second image on recognition. Furthermore, a binary classification operation is performed on the target feature matrix obtained from the fully connected processing to calculate object modality values. These modality values ​​are then used to determine the virtual objects that need to be identified and labeled in the second image, and bounding boxes are used to label the regions of virtual objects in the second image.

[0287] Steps 205-209 can be repeated in a step-by-step manner until the virtual object stops appearing in a certain frame of the video to be identified. Then the identification of the virtual object in the entire video to be identified is completed, and step 210 is executed.

[0288] 210. The server identifies virtual objects in the video to be identified based on the target detection box in the second image.

[0289] It should be noted that for virtual objects in motion within the video to be identified, the identification process can be understood as tracking and identifying the virtual object based on its motion trajectory. Specifically, the target motion trajectory corresponding to the virtual object can be generated based on the target detection box in the second image; the virtual object is then identified based on the motion trajectory.

[0290] To facilitate understanding of the embodiments of this application, specific application scenario examples will be used to describe the embodiments of this application. Specifically, by performing the above steps 201-210, and in conjunction with... Figures 4-9This document describes an example of this application scenario. Taking the virtual object recognition process in game videos as an example, it is applicable to various game types, such as first-person shooter (FPS) games, fantasy hero games, Clash Royale games, and multiplayer games. The specific description of this virtual object recognition scenario example is as follows:

[0291] I. Architecture of the Virtual Object Recognition System

[0292] The architecture of this virtual object recognition system includes a cloud server and a terminal. The cloud server and the user terminal are connected via a service transport network to achieve communication and data exchange. For example, the terminal can send an object recognition task and the game video to be recognized to the cloud server. It should be noted that when sending the game video, relevant information such as the game session information and identifiers can be sent. Then, the server can determine the virtual player characters to be recognized in the game video based on the object recognition task, and annotate them. The server can then track and recognize the virtual player characters based on the annotated target detection boxes, and return the recognition results to the terminal. For more information on the architecture of this virtual object recognition system, please refer to [link to relevant documentation]. Figure 4 As shown.

[0293] II. Description of the scenarios for this virtual object recognition method

[0294] See Figure 5 Taking a game scenario as an example, in order to identify virtual player characters in a game video, the scenario process of this virtual object recognition method may include:

[0295] 501. Obtain the game video to be identified and determine the virtual player character to be identified in the game video.

[0296] 502. Extract the first frame of the game video containing the virtual player character, and mark the position of the virtual player character in the first frame of the game video to obtain the game screen containing the detected virtual player character. Use the game screen containing the detected virtual player character as the previous frame of the game video.

[0297] 503. Take the previous frame of the game screen as the first game screen, and select the next frame of the game screen that is adjacent to the first game screen as the second game screen.

[0298] 504. Input the first game screen and the second game screen into the target model to obtain the target detection box of the virtual player character in the second game screen; wherein, the target model can label the position area of ​​the virtual player character in the second game screen.

[0299] 505. Based on the target detection box in the second game screen, track and identify the movement trajectory of the virtual player character in the game video.

[0300] This is so that the gameplay process of the virtual player character can be determined based on the tracking and identification results. For details regarding game screenshots containing object detection boxes, please refer to... Figure 6 As shown.

[0301] It is understandable that in step 504, the identification and labeling process of the virtual player character is achieved through the target model. For ease of understanding, this can be combined with... Figure 7 Let's understand the process by which the target model identifies and labels virtual player characters in the second game screen. Specifically:

[0302] The framework of this target model can include a feature extraction network, a feature aggregation network, and a classification network. The feature extraction network dynamically learns and extracts features of the specific virtual player character's location distribution in an image through multiple convolutional layers using a spatial awareness mechanism. The feature aggregation network integrates these extracted features and then adaptively learns channel weights through a channel awareness mechanism before aggregating them. The classification network distinguishes between the virtual player character and background information to label the virtual player character's location region. The following description, combining these networks, will illustrate the virtual player character recognition scenario.

[0303] (1) Feature extraction network incorporating spatial awareness mechanism:

[0304] See Figure 7 The feature extraction network comprises two sub-networks, defined as the first feature extraction sub-network A1 and the second feature extraction sub-network A2. These two sub-networks have identical structures and are parallel. Specifically, taking the first feature extraction sub-network A1 as an example, its structure includes a first convolutional layer (A1.1), a second convolutional layer (A1.2), a third convolutional layer (A1.3), a first spatial perception module (A1.4), a second spatial perception module (A1.5), a first feature fusion module (A1.6), and a second feature fusion module (A1.7). Spatial perception weights are obtained through the first spatial perception module (A1.4) and the second spatial perception module (A1.5) in the above feature value sub-networks. Then, the corresponding feature fusion modules are used to fuse the features of the image to enhance the feature representation of the virtual player character, and the corresponding convolutional layers are used to extract the features of the virtual player character.

[0305] Specifically, for each feature extraction subnetwork, the feature extraction process can be represented as follows:

[0306] F1 m=Conv(input),

[0307] F2 m =Conv(F1 m +TB(F1 m )),

[0308] F m =Conv(F2) m +TB(F2 m )).

[0309] Where Conv represents a convolutional layer, input represents the input image (such as the first image or the second image), TB represents spatial awareness, and F represents spatial awareness. m The output features (such as region feature matrix or color feature matrix) represent the m-modal of the image.

[0310] See Figure 8 The first spatial perception module (A1.4) and the second spatial perception module (A1.5) have the same structure, which may include: a first convolutional sub-unit, a rectified linear unit (ReLU), a local response normalization unit (LRN), a second convolutional sub-unit, and a classification unit (sigmoid function). The perception process of the spatial perception module can be represented as follows:

[0311] F 1 =LRN(ReLU(Conv(input))),

[0312] F output =Sigmoid(Conv(F 1 )).

[0313] Where Conv represents a convolutional layer, ReLU is a rectified linear unit, LRN is a local response normalization unit, Sigmoid is the sigmoid function, and Conv(input) and F output It is the input and output of the spatial perception module.

[0314] (2) Feature aggregation network including channel-aware mechanism:

[0315] See Figure 7 The feature aggregation network may include, in terms of structure, a channel-aware module (B.1), a first weighting module (B.2), a second weighting module (B.3), a first convolutional module (B.4), a second convolutional module (B.5), and a feature fusion module (B.6).

[0316] See Figure 9The channel sensing module (B.1) may include the following in terms of structure: feature connection module, fully connected layer (fc+fc), rectified linear unit (ReLU), dropout module, softmax function, and sigmoid function.

[0317] Specifically, the feature aggregation process performed by the feature aggregation network can be represented as follows:

[0318]

[0319] {F R 2, F T 2}=fc(fc(F concat )),

[0320] ωR, ωT = softmax({F R 2, F T 2}),

[0321]

[0322] in, Indicates a connection. Indicates channel weighting, F represents the process of element fusion. R F represents the output features of the feature extraction subnetwork for the second image. T This indicates that the feature extraction subnetwork outputs the features of the first image. `fc` corresponds to a fully connected layer, followed by ReLU and Dropout operations, yielding F... R 2 and F T 2, softmax represents the softmax function, ωT represents the first channel weights perceived through multimodal data related to the first image, ωR represents the second channel weights perceived through multimodal data related to the second image, and F represents the reconstructed target (about the virtual player character) description.

[0323] (3) The classification network consists of two fully connected layers (C.1) and a binary classifier (C.2).

[0324] Specifically, when identifying multiple virtual player characters in the same game frame, multiple binary classifiers (C.2) can be used after the two fully connected layers in the classification network to distinguish between the multiple virtual player characters. For each binary classifier, results that do not belong to its assigned modality are classified as negative samples, so that positive samples belong only to the virtual player character corresponding to that binary classifier; thus, multiple binary classifiers can identify multiple virtual player characters. The representation of each binary classifier is as follows:

[0325]

[0326] Where W is the feature classification weight, F is the fused input feature, net is the classifier output, and Q is the constant bias matrix.

[0327] Specifically, the update of weight W can be represented as follows:

[0328]

[0329] Among them, a i W represents the scaling factor (scaling parameter in the Dropout process) from the feedforward neural network, calculated as W. i J represents the feature classification weight, and J represents the neuron.

[0330] Through the above application scenarios, the following effects can be achieved: good recognition effect in game screens with multiple virtual player characters; high accuracy in distinguishing virtual player characters from background information in the game screen; effective segmentation of virtual player characters from the game screen; and automated tracking and recognition of virtual player characters.

[0331] As can be seen from the above, the embodiments of this application can determine the virtual object to be tracked and identified in the video to be identified, and extract a first image to determine the position of the virtual object and a second image to be identified based on the relationship between adjacent frames from the video to be identified. The first and second images are then enhanced with feature representation in the spatial and channel dimensions using a target model to highlight the shape of the virtual object in the image, making the virtual object more prominent relative to other backgrounds in the image. Furthermore, the virtual object features extracted from the first and second images are fused, and the virtual object to be identified in the second image is determined based on the aggregated feature matrix. A target detection box is then generated so that the virtual object can be identified based on the motion trajectory corresponding to a series of target detection boxes. In this way, the same virtual object can be identified and labeled sequentially from each image frame, effectively avoiding the influence of the variability of virtual objects in the image on the recognition process, and improving the efficiency and accuracy of virtual object recognition.

[0332] To facilitate understanding of this application, we can further describe it using an application scenario example of virtual object recognition in cloud gaming, in conjunction with steps 201-210 above. In this cloud gaming scenario, a cloud video stream of the game is generated, and the positions of virtual objects within the cloud video stream can be identified. The specific description of this virtual object recognition scenario example is as follows:

[0333] Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing. It enables thin clients with relatively limited graphics processing and data processing capabilities to run high-quality games. In cloud gaming, the game does not reside on the player's terminal but runs on a cloud server. The cloud server renders the game scene as a cloud video stream, which is then transmitted to the player's terminal via the network. The player's terminal does not need powerful graphics processing and data processing capabilities; it only needs basic streaming media playback capabilities and the ability to receive player input commands and send them to the cloud server.

[0334] Specifically, the process for recognizing virtual objects in the cloud video stream of this cloud game is as follows:

[0335] (a) Obtain the cloud video stream of the cloud game and identify the virtual objects to be identified in the cloud video stream.

[0336] (b) Extract the first frame game image containing the virtual object from the cloud video stream, and mark the position of the virtual object in the first frame game image to obtain the game image containing the identified virtual object with the detection box, and use the game image containing the identified virtual object as the previous frame game image.

[0337] (c) Take the previous frame game image as the first game image, and select the next frame game image that is adjacent to the first game image as the second game image.

[0338] (d) Input the first game image and the second game image into the target model to obtain the target detection box of the virtual object in the second game image; wherein, the target model can label the location region of the virtual object in the second game image.

[0339] (e) Based on the target detection box in the second game image, track and identify the motion trajectory of the virtual object in the cloud video stream.

[0340] This allows for the identification of virtual objects from the cloud video stream, enabling the feedback of their positions to the corresponding game client. Based on the virtual object's position within each frame of the game image in the cloud video stream, the game client generates a layout strategy for the game's virtual controls (buttons). This strategy allows for subsequent adjustments to the positions of the virtual buttons on the game client interface. This effectively avoids conflicts between the constantly changing positions of virtual objects and the positions of virtual game buttons (such as overlap or excessive proximity), ensuring a consistent distance between them and guaranteeing that the virtual objects are visible to the player, thus improving the player's gaming experience.

[0341] It is understandable that in step (e), the identification and annotation process of the virtual object is implemented through the target model. For ease of understanding, this can be combined with... Figures 7-9 Let's understand the process by which the target model identifies and annotates virtual objects in the second game image. It should be noted that in this cloud gaming scenario, the process of identifying virtual objects in the cloud video stream is equivalent to the aforementioned description of the "virtual object identification method in game videos," which can be found in the previous description. Further details will not be elaborated here.

[0342] To facilitate understanding of this application, it can be further described using a game live streaming application scenario, in conjunction with steps 201-210 above. In this game live streaming scenario, a live video stream is generated, and the location of virtual objects within the live video stream can be identified. The specific process for this virtual object identification scenario is as follows:

[0343] A. Obtain the live video stream of the game live stream type and identify the virtual objects to be identified in the live video stream.

[0344] B. Extract the first frame of the live video stream containing the virtual object, and mark the position of the virtual object in the first frame of the live video stream to obtain the live image of the identified virtual object containing the detection box. Use the live image of the identified virtual object as the previous frame of the live video stream.

[0345] C. Take the previous live frame as the first live image, and select the next live frame that is adjacent to the first live image as the second live image.

[0346] D. Input the first live image and the second live image into the target model to obtain the target detection box of the virtual object in the second live image; wherein, the target model can label the location region of the virtual object in the second live image.

[0347] E. Based on the target detection box in the second live image, track and identify the motion trajectory of virtual objects in the live video stream of the game live streaming type.

[0348] In this way, the position of virtual objects can be identified from each frame of the live video stream, and the position of virtual objects in the live video stream can be fed back to the corresponding live streaming client. The live streaming client can mark the changes in the position of virtual objects between consecutive frames of live video images, and decorate the virtual objects on the live streaming client with expressions or marks such as halos, smiley faces, and hearts, or decorate the virtual objects with colors that meet the visual needs of the audience (such as clothing, colorful virtual object skins, etc.). This can not only satisfy the audience's color vision, but also help the audience to identify the virtual objects they like in the game live video stream, increase their attention to the virtual objects, and improve the audience's user experience.

[0349] It is understandable that in step E, the identification and annotation process of the virtual object is implemented through the target model. For ease of understanding, this can be combined with... Figures 7-9 Let's understand the process by which the target model identifies and labels virtual objects in live stream images. It should be noted that in this game live stream scenario, the process of identifying virtual objects in the live video stream is identical to the aforementioned description of "virtual object identification method in game videos," which can be found therein; further details will not be elaborated upon here.

[0350] To better implement the above methods, embodiments of this application also provide a virtual object recognition device. For example, such as... Figure 10 As shown, the virtual object recognition device 1000 may include an acquisition unit 1001, an extraction unit 1002, a recognition unit 1003, and a recognition unit 1004.

[0351] The acquisition unit 1001 is used to acquire the video to be recognized and to determine the virtual objects to be recognized in the video.

[0352] Extraction unit 1002 is used to extract a first image and a second image of adjacent frames from the video to be identified, wherein the first image contains a detection box of a virtual object;

[0353] The recognition unit 1003 is used to input the first image and the second image into the target model to obtain the target detection box of the virtual object in the second image;

[0354] The target model enhances and fuses the region features of the virtual object within the detection box in the first image and the object color features in the second image, so as to predict the target detection box of the virtual object in the second image based on the fused feature matrix.

[0355] The recognition unit 1004 is used to recognize virtual objects in the video to be recognized based on the target detection box in the second image.

[0356] In some implementations, the recognition unit 1003 is further configured to: input the first image and the second image into the target model; extract the region features of the virtual object from the first image through a spatial attention mechanism to obtain a region feature matrix; extract the object color features from the second image through a spatial attention mechanism to obtain a color feature matrix; perform channel attention enhancement representation on the region feature matrix and the color feature matrix and fuse them to obtain an aggregated feature matrix; and annotate the location region of the virtual object in the second image based on the aggregated feature matrix to obtain a target detection box.

[0357] In some implementations, the recognition unit 1003 is further configured to: perform convolution processing on the second image to obtain a second image feature matrix; determine color feature weights from the second image feature matrix based on a spatial attention mechanism, and perform attention-enhanced representation of the object color features in the second image feature matrix according to the color feature weights to obtain a color feature matrix.

[0358] In some embodiments, the recognition unit 1003 is further configured to: determine initial color feature weights from color features in the second image feature matrix based on a spatial attention mechanism; perform preliminary enhancement representation on object color features in the second image feature matrix according to the initial color feature weights to obtain an initial fusion feature matrix; perform convolution processing on the initial fusion feature matrix to obtain an initial color feature matrix; perform perceptual processing on color features in the initial color feature matrix through a spatial attention mechanism, and perform enhancement representation on object color features in the initial fusion feature matrix according to target color feature weights obtained from the perceptual processing to obtain a target fusion feature matrix; and perform convolution processing on the target fusion feature matrix to obtain a color feature matrix.

[0359] In some embodiments, the recognition unit 1003 is further configured to: perform convolution processing on the second image feature matrix based on a spatial attention mechanism to obtain a first convolution feature matrix; perform eigenvalue normalization processing on the first convolution feature matrix, and perform local response normalization processing on the eigenvalues ​​in the initial eigenvalue matrix obtained by the normalization processing to obtain a target eigenvalue matrix; perform convolution processing on the target eigenvalue matrix to obtain a second convolution feature matrix; and perform color weight classification on the color features in the second convolution feature matrix to obtain initial color feature weights.

[0360] In some embodiments, the recognition unit 1003 is further configured to: perform perceptual processing on the region feature matrix based on the channel attention mechanism to obtain the first channel weights, and perform channel attention-enhanced representation on the region feature matrix according to the first channel weights to obtain the first channel feature matrix; perform perceptual processing on the color feature matrix based on the channel attention mechanism to obtain the second channel weights, and perform channel attention-enhanced representation on the color feature matrix according to the second channel weights to obtain the second channel feature matrix; perform convolution processing on the first channel feature matrix and the second channel feature matrix respectively to obtain the target region feature matrix and the target color feature matrix; and fuse the target region feature matrix and the target color feature matrix in the channels to obtain the aggregated feature matrix.

[0361] In some implementations, the recognition unit 1003 is further configured to: perform fully connected processing on the color feature matrix based on the channel attention mechanism to obtain a first feature matrix; hide some feature values ​​in the first feature matrix according to feature compliance to obtain a second feature matrix; and integrate the color weights of the second feature matrix through an activation function to obtain a second channel weight.

[0362] In some implementations, the recognition unit 1003 is used to: perform fully connected processing on the aggregated feature matrix to obtain a target feature matrix; perform binary classification processing on the target feature matrix to obtain an object modality value; and annotate the location region of the corresponding virtual object in the second image according to the object modality value to obtain a target detection box.

[0363] In some implementations, the identification unit 1003 is further configured to: obtain the feature hiding parameters of the target model in the channel attention-enhanced representation, and calculate the feature classification weights based on the feature hiding parameters; perform weighted processing on the target feature matrix based on the feature classification weights to obtain the target weight feature matrix; and accumulate the matrix difference between the target weight feature matrix and the constant bias matrix to obtain the object modality value.

[0364] In some implementations, the recognition unit 1003 is further configured to: perform fully connected processing on the aggregated feature matrix to obtain a target feature matrix containing features of multiple virtual objects; perform binary classification processing on the target feature matrix through different binary classification modules in the target model to obtain the object modality value corresponding to each virtual object feature; and annotate the location region of each virtual object in the second image according to the object modality value to obtain the target detection box corresponding to each virtual object.

[0365] In some embodiments, the virtual object recognition device further includes a training unit, configured to: acquire a first sample image and a second sample image of adjacent frame relationships, wherein the first sample image and the second sample image contain the same sample virtual object, and the first sample image contains an initial sample detection box of the sample virtual object; train a preset model by using the first sample image and the second sample image as input and the initial sample detection box as training labels; until the predicted detection box in the second sample image output by the preset model is consistent with the sample virtual object labeled by the initial sample detection box, thereby obtaining the trained target model.

[0366] In some implementations, the extraction unit 1002 is further configured to: extract the first frame image containing the virtual object to be identified from the video to be identified; and annotate the virtual object to be identified in the first frame image to obtain a target image frame containing the detection box of the annotated virtual object.

[0367] The extraction unit 1002 is further configured to use the target image frame as the previous frame image and determine the previous frame image as the first image; according to the adjacent frame relationship in the video frame to be identified, obtain the next image frame adjacent to the first image and determine the next image frame as the second image.

[0368] As can be seen from the above, the embodiments of this application can determine the virtual object to be tracked and identified in the video to be identified, and extract a first image to determine the position of the virtual object and a second image to be identified based on the relationship between adjacent frames from the video to be identified. The first and second images are then enhanced with feature representation in the spatial and channel dimensions using a target model to highlight the shape of the virtual object in the image, making the virtual object more prominent relative to other backgrounds in the image. Furthermore, the virtual object features extracted from the first and second images are fused, and the virtual object to be identified in the second image is determined based on the aggregated feature matrix. A target detection box is then generated so that the virtual object can be identified based on the motion trajectory corresponding to a series of target detection boxes. In this way, the same virtual object can be identified and labeled sequentially from each image frame, effectively avoiding the influence of the variability of virtual objects in the image on the recognition process, and improving the efficiency and accuracy of virtual object recognition.

[0369] This application also provides a computer device, such as... Figure 11 As shown, it illustrates a structural schematic diagram of the computer device involved in the embodiments of this application, specifically:

[0370] The computer device may include components such as a processor 1101 with one or more processing cores, a memory 1102 with one or more computer-readable storage media, a power supply 1103, and an identification unit 1104. Those skilled in the art will understand that... Figure 11The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0371] The processor 1101 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the processor 1101 may include one or more processing cores; preferably, the processor 1101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1101.

[0372] The memory 1102 can be used to store software programs and modules. The processor 1101 executes various functional applications and virtual object recognition by running the software programs and modules stored in the memory 1102. The memory 1102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1102 may also include a memory controller to provide the processor 1101 with access to the memory 1102.

[0373] The computer device also includes a power supply 1103 that supplies power to the various components. Preferably, the power supply 1103 can be logically connected to the processor 1101 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 1103 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0374] The computer device may also include an identification unit 1104, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0375] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in the embodiments of this application, the processor 1101 in the computer device loads the executable files corresponding to the processes of one or more applications into the memory 1102 according to the following instructions, and the processor 1101 runs the applications stored in the memory 1102 to realize various functions, as follows:

[0376] The process involves: acquiring a video to be identified and identifying virtual objects within it; extracting a first image and a second image from adjacent frames of the video, where the first image contains the detection bounding box of the virtual object; inputting the first and second images into a target model to obtain the target detection bounding box of the virtual object in the second image; wherein the target model enhances and fuses the region features of the virtual object within the detection bounding box in the first image and the object color features in the second image to predict the target detection bounding box of the virtual object in the second image based on the fused aggregated feature matrix; and identifying the virtual object in the video based on the target detection bounding box in the second image.

[0377] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0378] Therefore, this scheme can identify the virtual object to be tracked and identified in the video to be identified, and extract a first image to determine the location of the virtual object and a second image to be identified based on the relationship between adjacent frames from the video to be identified. The first and second images are then enhanced with feature representations in the spatial and channel dimensions using a target model to highlight the shape of the virtual object in the image, making the virtual object more prominent relative to other background elements. Furthermore, the virtual object features extracted from the first and second images are fused, and the virtual object to be identified in the second image is determined based on the aggregated feature matrix. Target detection boxes are then generated to identify the virtual object based on the motion trajectory corresponding to a series of target detection boxes. In this way, the same virtual object can be identified and labeled sequentially from each image frame, effectively avoiding the influence of the variability of virtual objects in the image on the recognition process, and improving the efficiency and accuracy of virtual object recognition.

[0379] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0380] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the virtual object recognition methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0381] The process involves: acquiring a video to be identified and identifying virtual objects within it; extracting a first image and a second image from adjacent frames of the video, where the first image contains the detection bounding box of the virtual object; inputting the first and second images into a target model to obtain the target detection bounding box of the virtual object in the second image; wherein the target model enhances and fuses the region features of the virtual object within the detection bounding box in the first image and the object color features in the second image to predict the target detection bounding box of the virtual object in the second image based on the fused aggregated feature matrix; and identifying the virtual object in the video based on the target detection bounding box in the second image.

[0382] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0383] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0384] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the virtual object recognition methods provided in the embodiments of this application, the beneficial effects that any of the virtual object recognition methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0385] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0386] The foregoing has provided a detailed description of a virtual object recognition method, apparatus, device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for recognizing virtual objects, characterized in that, include: Acquire the video to be identified and determine the virtual object to be identified in the video; Extract a first image and a second image of adjacent frames from the video to be identified, wherein the first image contains the detection box of the virtual object; The first image and the second image are input into the target model; the region features of the virtual object are extracted from the first image through a spatial attention mechanism to obtain the region feature matrix; The spatial attention mechanism is used to extract object color features from the second image to obtain a color feature matrix; the region feature matrix and the color feature matrix are then represented by channel attention enhancement and fused to obtain an aggregated feature matrix. Based on the aggregated feature matrix, the location regions of the virtual objects in the second image are labeled to obtain the target detection boxes of the virtual objects in the second image; Based on the target detection box in the second image, the virtual objects in the video to be identified are identified.

2. The method according to claim 1, characterized in that, The step of extracting object color features from the second image using the spatial attention mechanism to obtain a color feature matrix includes: The second image is convolved to obtain the feature matrix of the second image; Based on the spatial attention mechanism, color feature weights are determined from the second image feature matrix, and the object color features in the second image feature matrix are enhanced with attention according to the color feature weights to obtain the color feature matrix.

3. The method according to claim 2, characterized in that, The step of determining color feature weights from the second image feature matrix based on the spatial attention mechanism, and performing attention-enhanced representation of object color features in the second image feature matrix according to the color feature weights to obtain a color feature matrix includes: Based on the spatial attention mechanism, the initial color feature weights are determined from the color features in the second image feature matrix; Based on the initial color feature weights, the object color features in the second image feature matrix are initially enhanced to obtain an initial fusion feature matrix; The initial fusion feature matrix is ​​convolved to obtain the initial color feature matrix; The spatial attention mechanism is used to perform perceptual processing on the color features in the initial color feature matrix, and the object color features in the initial fusion feature matrix are enhanced based on the target color feature weights obtained from the perceptual processing to obtain the target fusion feature matrix. The target fusion feature matrix is ​​convolved to obtain the color feature matrix.

4. The method according to claim 3, characterized in that, The method of determining initial color feature weights from color features in the second image feature matrix based on spatial attention mechanism includes: Based on the spatial attention mechanism, the second image feature matrix is ​​convolved to obtain the first convolutional feature matrix; The first convolutional feature matrix is ​​subjected to eigenvalue normalization, and the eigenvalues ​​in the initial eigenvalue matrix obtained by normalization are subjected to local response normalization to obtain the target eigenvalue matrix. The target feature value matrix is ​​convolved to obtain the second convolution feature matrix; Color features in the second convolutional feature matrix are classified by color weights to obtain initial color feature weights.

5. The method according to any one of claims 1-4, characterized in that, The step of performing channel attention-enhanced representation and fusing the region feature matrix and the color feature matrix to obtain an aggregated feature matrix includes: The region feature matrix is ​​perceptually processed based on the channel attention mechanism to obtain the first channel weight, and the region feature matrix is ​​then represented by channel attention enhancement based on the first channel weight to obtain the first channel feature matrix. Based on the channel attention mechanism, the color feature matrix is ​​perceptually processed to obtain the second channel weights, and the color feature matrix is ​​then represented by channel attention enhancement based on the second channel weights to obtain the second channel feature matrix. The first channel feature matrix and the second channel feature matrix are convolved respectively to obtain the target region feature matrix and the target color feature matrix. The target region feature matrix and the target color feature matrix are fused together in the channel to obtain the aggregated feature matrix.

6. The method according to claim 5, characterized in that, The process of performing perceptual processing on the color feature matrix based on the channel attention mechanism to obtain the second channel weights includes: Based on the channel attention mechanism, the color feature matrix is ​​processed by a fully connected layer to obtain the first feature matrix; According to the feature compliance, some feature values ​​in the first feature matrix are hidden to obtain the second feature matrix; The second channel weights are obtained by integrating the color weights of the second feature matrix through an activation function.

7. The method according to claim 1, characterized in that, The step of annotating the location regions of virtual objects in the second image based on the aggregated feature matrix to obtain target detection boxes includes: The aggregated feature matrix is ​​fully connected to obtain the target feature matrix. The target feature matrix is ​​subjected to binary classification to obtain the object modality value; Based on the object modality value, the location region of the corresponding virtual object in the second image is marked to obtain the target detection box.

8. The method according to claim 7, characterized in that, The binary classification process of the target feature matrix to obtain object modality values ​​includes: Obtain the feature hiding parameters of the target model in the channel attention-enhanced representation, and calculate the feature classification weights based on the feature hiding parameters; The target feature matrix is ​​weighted according to the feature classification weights to obtain the target weight feature matrix; The matrix difference between the target weight feature matrix and the constant bias matrix is ​​accumulated to obtain the object modality value.

9. The method according to claim 1, characterized in that, The aggregated feature matrix contains multiple different virtual object features. Based on the aggregated feature matrix, the positions of the virtual objects in the second image are labeled to obtain target detection boxes, including: The aggregated feature matrix is ​​fully connected to obtain a target feature matrix containing features of multiple virtual objects; The target feature matrix is ​​subjected to binary classification processing by different binary classification modules in the target model to obtain the object modality value corresponding to each virtual object feature; Based on the object modality value, the location region of each virtual object in the second image is labeled to obtain the target detection box corresponding to each virtual object.

10. The method according to claim 1, characterized in that, Before extracting the first and second images of adjacent frames from the video to be identified, the method further includes: Obtain a first sample image and a second sample image representing the relationship between adjacent frames. The first sample image and the second sample image contain the same virtual sample object. The first sample image contains the initial sample detection box of the virtual sample object. The first and second sample images are used as input, and the initial sample detection boxes are used as training labels to train the preset model. The training of the target model continues until the predicted detection box in the second sample image output by the preset model matches the virtual sample object labeled in the initial sample detection box.

11. The method according to claim 1, characterized in that, Before extracting the first and second images of adjacent frames from the video to be identified, the method further includes: Extract the first frame image containing the virtual object to be identified from the video to be identified; The virtual objects to be identified in the first frame image are labeled to obtain a target image frame containing the detection boxes labeled with the virtual objects; The step of extracting the first and second images of adjacent frames from the video to be identified includes: The target image frame is taken as the previous frame image, and the previous frame image is determined as the first image; Based on the relationship between adjacent frames in the video to be identified, the next image frame adjacent to the first image is obtained, and the next image frame is determined as the second image.

12. A virtual object recognition device, characterized in that, include: An acquisition unit is used to acquire a video to be identified and to determine the virtual object to be identified in the video. An extraction unit is used to extract a first image and a second image of adjacent frames from the video to be identified, wherein the first image contains the detection box of the virtual object; The recognition unit is used to input the first image and the second image into the target model; and to extract the region features of the virtual object from the first image through a spatial attention mechanism to obtain a region feature matrix. The spatial attention mechanism is used to extract object color features from the second image to obtain a color feature matrix; the region feature matrix and the color feature matrix are then represented by channel attention enhancement and fused to obtain an aggregated feature matrix. Based on the aggregated feature matrix, the location regions of virtual objects in the second image are labeled to obtain target detection boxes; The recognition unit is further configured to recognize virtual objects in the video to be recognized based on the target detection box in the second image.

13. The apparatus according to claim 12, characterized in that, The recognition unit is further configured to: perform convolution processing on the second image to obtain a second image feature matrix; Based on the spatial attention mechanism, color feature weights are determined from the second image feature matrix, and the object color features in the second image feature matrix are enhanced with attention according to the color feature weights to obtain the color feature matrix.

14. The apparatus according to claim 13, characterized in that, The identification unit is also used for: Based on the spatial attention mechanism, the initial color feature weights are determined from the color features in the second image feature matrix; Based on the initial color feature weights, the object color features in the second image feature matrix are initially enhanced to obtain an initial fusion feature matrix; The initial fusion feature matrix is ​​convolved to obtain the initial color feature matrix; The spatial attention mechanism is used to perform perceptual processing on the color features in the initial color feature matrix, and the object color features in the initial fusion feature matrix are enhanced based on the target color feature weights obtained from the perceptual processing to obtain the target fusion feature matrix. The target fusion feature matrix is ​​convolved to obtain the color feature matrix.

15. The apparatus according to claim 14, characterized in that, The identification unit is further configured to: Based on the spatial attention mechanism, the second image feature matrix is ​​convolved to obtain the first convolutional feature matrix; The first convolutional feature matrix is ​​subjected to eigenvalue normalization, and the eigenvalues ​​in the initial eigenvalue matrix obtained by normalization are subjected to local response normalization to obtain the target eigenvalue matrix. The target feature value matrix is ​​convolved to obtain the second convolution feature matrix; Color features in the second convolutional feature matrix are classified by color weights to obtain initial color feature weights.

16. The apparatus according to any one of claims 12-15, characterized in that, The identification unit is further configured to: The region feature matrix is ​​perceptually processed based on the channel attention mechanism to obtain the first channel weight, and the region feature matrix is ​​then represented by channel attention enhancement based on the first channel weight to obtain the first channel feature matrix. Based on the channel attention mechanism, the color feature matrix is ​​perceptually processed to obtain the second channel weights, and the color feature matrix is ​​then represented by channel attention enhancement based on the second channel weights to obtain the second channel feature matrix. The first channel feature matrix and the second channel feature matrix are convolved respectively to obtain the target region feature matrix and the target color feature matrix. The target region feature matrix and the target color feature matrix are fused together in the channel to obtain the aggregated feature matrix.

17. The apparatus according to claim 16, characterized in that, The identification unit is further configured to: Based on the channel attention mechanism, the color feature matrix is ​​processed by a fully connected layer to obtain the first feature matrix; According to the feature compliance, some feature values ​​in the first feature matrix are hidden to obtain the second feature matrix; The second channel weights are obtained by integrating the color weights of the second feature matrix through an activation function.

18. The apparatus according to claim 12, characterized in that, The identification unit is further configured to: The aggregated feature matrix is ​​fully connected to obtain the target feature matrix. The target feature matrix is ​​subjected to binary classification to obtain the object modality value; Based on the object modality value, the location region of the corresponding virtual object in the second image is marked to obtain the target detection box.

19. The apparatus according to claim 18, characterized in that, The identification unit is further configured to: obtain the feature hiding parameters of the target model in the channel attention-enhanced representation, and calculate the feature classification weights based on the feature hiding parameters; The target feature matrix is ​​weighted according to the feature classification weights to obtain the target weight feature matrix; The matrix difference between the target weight feature matrix and the constant bias matrix is ​​accumulated to obtain the object modality value.

20. The apparatus according to claim 12, characterized in that, The aggregated feature matrix contains multiple different virtual object features, and the recognition unit is further configured to: perform fully connected processing on the aggregated feature matrix to obtain a target feature matrix containing multiple virtual object features; The target feature matrix is ​​subjected to binary classification processing by different binary classification modules in the target model to obtain the object modality value corresponding to each virtual object feature; Based on the object modality value, the location region of each virtual object in the second image is labeled to obtain the target detection box corresponding to each virtual object.

21. The apparatus according to claim 12, characterized in that, The virtual object recognition device further includes a training unit, used to: acquire a first sample image and a second sample image of adjacent frame relationships, wherein the first sample image and the second sample image contain the same sample virtual object, and the first sample image contains the initial sample detection box of the sample virtual object; The first and second sample images are used as input, and the initial sample detection boxes are used as training labels to train the preset model. The training of the target model continues until the predicted detection box in the second sample image output by the preset model matches the virtual sample object labeled in the initial sample detection box.

22. The apparatus according to claim 12, characterized in that, The extraction unit is also used for: Extract the first frame image containing the virtual object to be identified from the video to be identified; The virtual objects to be identified in the first frame image are labeled to obtain a target image frame containing the detection boxes labeled with the virtual objects; The target image frame is taken as the previous frame image, and the previous frame image is determined as the first image; Based on the relationship between adjacent frames in the video to be identified, the next image frame adjacent to the first image is obtained, and the next image frame is determined as the second image.

23. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor running the computer program in the memory to implement the steps of the virtual object recognition method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the virtual object recognition method according to any one of claims 1 to 11.

25. A computer program product comprising computer instructions, wherein the computer instructions, when executed, implement the steps of the virtual object recognition method according to any one of claims 1 to 11.

Images