Methods, devices, equipment, media, and program products for identifying virtual objects

Abstract

This application discloses a method, apparatus, device, medium, and program product for identifying virtual objects, belonging to the field of virtual environments. The method includes: acquiring a first image from a video stream based on a virtual environment; marking a first region within the first image based on frequency domain information of the first image, the first region corresponding to a first virtual object; acquiring a second image from the video stream based on the virtual environment; marking a second region within the second image based on frequency domain information of the second image, the second region corresponding to a second virtual object; and determining that the first virtual object and the second virtual object are the same virtual object based on image features of the first and second regions. This solution not only supports identifying the same virtual object but also determines the movement trajectory of the same virtual object.

Description

Technical Field

[0001] This application relates to the field of virtual environments, and in particular to a method, apparatus, device, medium, and program product for identifying virtual objects. Background Technology

[0002] In the gaming industry, it is often necessary to identify virtual objects within the game.

[0003] In related technologies, the frame difference method is used to detect virtual objects in games. In two adjacent frames, the background of the image moves slowly, while the virtual object moves faster. By taking the difference between the next frame and the previous frame, the virtual object can be detected.

[0004] However, when multiple virtual objects exist in two adjacent frames, the technology can only detect the existence of multiple virtual objects, but cannot distinguish between them. In other words, it cannot make a one-to-one correspondence between the multiple virtual objects in the later frame and the multiple virtual objects in the previous frame. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program product for identifying virtual objects, which not only supports identifying the same virtual object but also determines the movement trajectory of the same virtual object. The technical solution is as follows:

[0006] According to one aspect of this application, a method for identifying virtual objects is provided, the method comprising:

[0007] A first image is acquired from a video stream based on a virtual environment; based on the frequency domain information of the first image, a first region is marked within the first image, and the first region corresponds to a first virtual object;

[0008] A second image is acquired from a video stream based on a virtual environment; based on the frequency domain information of the second image, a second region is marked within the second image, and the second region corresponds to a second virtual object;

[0009] Based on the image features of the first region and the second region, the first virtual object and the second virtual object are determined to be the same virtual object.

[0010] According to another aspect of this application, a virtual object identification device is provided, the device comprising:

[0011] An acquisition module is used to acquire a first image from a video stream based on a virtual environment;

[0012] The marking module is used to mark a first region in the first image based on the frequency domain information of the first image, and the first region corresponds to a first virtual object;

[0013] The acquisition module is also used to acquire a second image from a video stream based on a virtual environment;

[0014] The marking module is also used to mark a second region within the second image based on the frequency domain information of the second image, and the second region corresponds to a second virtual object;

[0015] The determination module is used to determine that the first virtual object and the second virtual object are the same virtual object based on the image features of the first region and the second region.

[0016] According to one aspect of this application, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the virtual object identification method as described above.

[0017] According to another aspect of this application, a computer-readable storage medium is provided, the storage medium storing a computer program that is loaded and executed by a processor to implement the virtual object identification method as described above.

[0018] According to another aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the virtual object identification method provided in the above aspect.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] A first region corresponding to a first virtual object is marked based on the frequency domain information of the first image, and a second region corresponding to a second virtual object is marked based on the frequency domain information of the second image. Then, based on the image features of the first and second regions, it is determined that the first and second virtual objects are the same virtual object. This provides a method for identifying virtual objects. This method not only supports the identification of virtual objects in image frames, but also supports the one-to-one correspondence between virtual objects in two image frames to obtain the movement trajectory of the virtual object. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1A block diagram of a computer system provided in an exemplary embodiment is shown;

[0023] Figure 2 A flowchart of a virtual object identification method provided by an exemplary embodiment is shown;

[0024] Figure 3 A schematic diagram of a virtual object identification method provided by an exemplary embodiment is shown;

[0025] Figure 4 A schematic diagram of a compressor provided in an exemplary embodiment is shown;

[0026] Figure 5 A schematic diagram of a decompressor provided in an exemplary embodiment is shown;

[0027] Figure 6 A flowchart of a virtual object identification method provided by another exemplary embodiment is shown;

[0028] Figure 7 A schematic diagram of a virtual object identification method provided by another exemplary embodiment is shown;

[0029] Figure 8 A structural block diagram of a virtual object recognition device provided in an exemplary embodiment is shown;

[0030] Figure 9 A structural block diagram of a computer device provided in an exemplary embodiment is shown. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] It should be understood that "several" in this article refers to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0033] First, a brief introduction to the terms used in the embodiments of this application:

[0034] Virtual environment: This refers to the virtual environment displayed (or provided) by the client when running on the terminal. This virtual environment can be a simulation of the real world, a semi-simulated / semi-fictional environment, or a purely fictional environment. The virtual environment can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual environment; this application does not limit it to any particular type. The following embodiments use a three-dimensional virtual environment as an example.

[0035] Optionally, the virtual environment can provide a battle environment for virtual objects. For example, in a MOBA (Multiplayer Online Battle Arena) game, two opposing factions battle in a virtual environment, with the side that destroys the enemy faction's defensive virtual machine level winning. For example, in a battle royale game, at least one virtual object engages in a single match within the virtual environment. The virtual object survives by avoiding attacks from enemy units and dangers present in the virtual environment (such as poison gas circles, swamps, etc.). When a virtual object's health reaches zero, its life ends, and the last surviving virtual object is the winner.

[0036] Virtual objects refer to movable objects in a virtual environment. These movable objects can be virtual characters, virtual animals, anime characters, etc., such as people and animals displayed in a 3D virtual environment. Optionally, virtual objects are 3D models created based on animation skeletal technology. Each virtual object has its own shape and volume in the 3D virtual environment and occupies a portion of the space in the 3D virtual environment.

[0037] Figure 1 A structural block diagram of a computer system provided in an exemplary embodiment of this application is shown. The computer system 100 includes a terminal 120 and a server 140.

[0038] Terminal 120 has a client installed and running that supports virtual environments. The client can be any of the following: a 3D map program, a side-scrolling shooter, a side-scrolling adventure, a side-scrolling platformer, a side-scrolling strategy game, a virtual reality (VR) application, or an augmented reality (AR) application.

[0039] Terminal 120 is connected to server 140 via a wireless network or a wired network.

[0040] Server 140 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Exemplarily, server 140 includes a processor 144 and a memory 142. Memory 142 further includes a receiving module 1421, a control module 1422, and a sending module 1423. The receiving module 1421 receives requests sent by clients, such as requests to control the movement of virtual objects. The control module 1422 controls the rendering of the virtual environment. The sending module 1423 sends responses to clients, such as indicating that the position of a virtual object has changed. Server 140 provides background services for applications supporting the virtual environment. Optionally, server 140 performs the primary computing task, and terminal 120 performs secondary computing tasks; or, server 140 performs secondary computing tasks, and terminal 120 performs primary computing tasks; or, server 140 and terminal 120 perform computing tasks collaboratively.

[0041] Optionally, the aforementioned client can run on different operating system platforms (Android or iOS). Optionally, the terminal device type includes at least one of the following: smartphone, smartwatch, in-vehicle terminal, wearable device, smart TV, tablet computer, e-book reader, MP3 player, MP4 player, laptop computer, and desktop computer.

[0042] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number of terminals or the type of device.

[0043] Figure 2 A flowchart illustrating a virtual object identification method provided in an exemplary embodiment of this application is shown. The method is... Figure 1 The terminal or server shown is used to illustrate the example, and the method includes:

[0044] Step 210: Acquire the first image from the video stream based on the virtual environment;

[0045] The first image refers to any single image in the video stream of the virtual environment. In one embodiment, the video stream of the virtual environment can be the video displayed on the user terminal's screen. The user terminal acquires the virtual environment image by calling a camera model and displays the virtual environment image on the screen. Optionally, the video stream of the virtual environment can also be the virtual environment image acquired by the server through the camera model in the virtual environment. Optionally, the interval between adjacent image frames in the video stream of the virtual environment is a preset interval. For example, the virtual environment image acquired through the camera model every 2 seconds can be used as the video stream.

[0046] Reference Figure 3This shows the first image 311 acquired from a video stream based on a virtual environment. Figure 3 It can be seen that the first image 311 contains two virtual objects: a "cannon" and a "soldier".

[0047] Step 220: Based on the frequency domain information of the first image, mark the first region within the first image;

[0048] In one embodiment, the frequency domain information of the first image can be understood as the spectrum obtained by performing a Fourier transform on the first image. The Fourier transform of the image is used to transform the gray-level distribution of the image into the frequency distribution of the image. The high-frequency components in the spectrum information correspond to the detail information of the image, while the low-frequency components correspond to the contour information of the image. Furthermore, the high-frequency components represent the abrupt changes in the signal (i.e., large gray-level gradients), while the low-frequency components determine the overall image of the signal (i.e., small gradients).

[0049] In one embodiment, the terminal or server obtains the frequency domain information of the first image by performing a DCT (Discrete Cosine Transform); inputs the frequency domain information of the first image into a high-pass filter for recognizing virtual objects to obtain a first DCT matrix, which stores information about the first virtual object; and marks a first region within the first image based on the first DCT matrix.

[0050] Optionally, step 220 can be performed by the following steps.

[0051] First, the first image is transformed using the following formula (1);

[0052]

[0053] Where f(x, y) are the pixels of the first image, F(u, v) are the frequency domain information of the first image after DCT transformation, and N is the number of pixels.

[0054] Then, the frequency domain information of the first image obtained by formula (1) is passed through the high-pass filter of formula (2) to obtain the first DCT matrix.

[0055]

[0056] Wherein, P and Q are preset frequency bands. Optionally, the region where the virtual object is located is extracted from a large number of images offline, and the frequency information of the region is extracted. By statistically analyzing the frequency domain information of a large number of regions, a comprehensive frequency band is obtained, and the critical points of the comprehensive frequency band are P and Q.

[0057] Next, after obtaining the first DCT matrix through a high-pass filter, an attention mechanism is used to assign first attention weights to the matrix blocks in the first DCT matrix that indicate the first region, resulting in a weighted first DCT matrix.

[0058] Attention mechanisms are essentially similar to the selective visual attention mechanisms of humans, used to select core information relevant to the current task objective from a large amount of information. In other words, attention mechanisms are used to adjust the weight of core information and increase its weight.

[0059] In one embodiment, the first DCT matrix corresponds to the pixel matrix of the first image. The first DCT matrix includes multiple matrix blocks. An attention mechanism is used to assign a first attention weight to the matrix blocks indicating the first region, resulting in a weighted first DCT matrix. Assigning a first attention weight to the matrix blocks indicating the first region can be understood as: reducing the attention weight of matrix blocks indicating other regions, and / or increasing the attention weight of matrix blocks indicating the first region.

[0060] Finally, based on the weighted first DCT matrix and the pixel matrix of the first image, the first region is marked within the first image.

[0061] In one embodiment, combining the weighted first DCT matrix with the pixel matrix of the first image can amplify the difference in pixel values ​​between pixels in the first region and pixels in other regions of the first image, thus distinguishing the first region from other regions.

[0062] In one embodiment, a first region is marked using a pre-defined procedure. Optionally, the first region is marked by adding a rectangular frame. For example, when distinguishing the first region from other regions, the program marks the boundaries of the first region with straight lines. After all lines are marked, the program finds the boxes formed by the lines at the boundaries and trims off the longer lines, thus adding a rectangular frame at the boundary of the first region. Optionally, when adding lines, the lines should pass through as many pixels as possible at the edge of the first region, so that the marked rectangular frame fits as closely as possible to the edge of the first region.

[0063] Optionally, the first region can be marked by adding an irregular bounding box. For example, when distinguishing the first region from other regions, the program directly concatenates the pixels at the edges of the first region, ultimately adding an irregular bounding box to the first region.

[0064] In summary, please refer to the following: Figure 3 Step 220 is... Figure 3The left half of the image. After the first image 311 undergoes DCT transformation and filtering, the first DCT matrix is ​​obtained. The first DCT matrix is ​​then input into the compressor and decompressor to mark the first region. Figure 3 The rectangle 312 in the image is the marker added to the first region. The first virtual object (soldier) exists within this first region.

[0065] Figure 3 The left half of the diagram also shows a recorder 313, which is used to record the type and position of virtual objects within the first image. Schematably, two circles with filled content in the recorder 313 represent two virtual objects (a cannon and a soldier) within the first image. Circles with different filled content represent different virtual objects. Furthermore, the positions of the two filled circles on the recorder 313 correspond to the positions of the two virtual objects in the first image; the circle on the left represents the virtual object on the left side of the first image, and the circle on the right represents the virtual object on the right side. This pattern continues, meaning that when there are more than two virtual objects in the first image, the recorder 313 records the virtual objects sequentially. Therefore, the recorder 313 can record the virtual objects within the first image, and by combining the recorders of adjacent images, the movement trajectory of any virtual object can be obtained.

[0066] Step 230: Acquire the second image from the video stream based on the virtual environment;

[0067] The second image refers to the image in the video stream of the virtual environment that follows the first image. (Refer to the reference.) Figure 3 This illustrates a second image 321 acquired from a video stream based on a virtual environment. Figure 3 It can be seen that the second image 321 contains two virtual objects, "cannon" and "soldier".

[0068] Step 240: Based on the frequency domain information of the second image, mark the second region within the second image;

[0069] In one embodiment, the terminal or server obtains the frequency domain information of the second image by performing a DCT transform on the second image; inputs the frequency domain information of the second image into a high-pass filter for identifying virtual objects to obtain a second DCT matrix, which stores information about the second virtual object; and marks a second region within the second image based on the second DCT matrix.

[0070] Optionally, step 240 can be performed by the following steps.

[0071] First, the second image is transformed using the following formula (3);

[0072]

[0073] Where f(x, y) are the pixels of the second image, F(u, v) are the frequency domain information of the second image after DCT transformation, and N is the number of pixels.

[0074] Then, the frequency domain information of the second image obtained by formula (3) is passed through the high-pass filter shown in formula (4) to obtain the second DCT matrix.

[0075]

[0076] Wherein, P and Q are preset frequency band boundaries. Optionally, the region where the virtual object is located is extracted offline from a large number of images, and the frequency information of the region is extracted. By statistically analyzing the frequency domain information of a large number of regions, a comprehensive frequency band is obtained, and the critical points of the comprehensive frequency band are P and Q.

[0077] Next, after obtaining the second DCT matrix through a high-pass filter, an attention mechanism is used to assign second attention weights to the matrix blocks in the second DCT matrix that indicate the second region, resulting in a weighted second DCT matrix.

[0078] In one embodiment, the second DCT matrix corresponds to the pixel matrix of the second image. The second DCT matrix includes multiple matrix blocks. A second attention weight is assigned to the matrix blocks indicating the second region through an attention mechanism to obtain a weighted second DCT matrix. Assigning a second attention weight to the matrix blocks indicating the second region can be understood as: reducing the attention weight of matrix blocks indicating other regions, and / or increasing the attention weight of matrix blocks indicating the second region.

[0079] Finally, based on the weighted second DCT matrix and the pixel matrix of the second image, the second region is marked within the second image.

[0080] In one embodiment, combining the weighted second DCT matrix with the pixel matrix of the second image can amplify the difference in pixel values ​​between pixels in the second region and pixels in other regions of the second image, thus distinguishing the second region from other regions.

[0081] In one embodiment, the second region is marked using a pre-defined procedure. Optionally, the second region is marked by adding a rectangular frame. For example, when distinguishing the second region from other regions, the program marks the boundaries of the second region with straight lines. After all lines are marked, the program identifies the boxes formed by the lines at the boundaries and trims off the longer lines, thus adding a rectangular frame at the boundary of the second region. Optionally, when adding lines, the lines should pass through as many pixels as possible at the edge of the second region, so that the marked rectangular frame fits as closely as possible to the edge of the second region.

[0082] Optionally, the second region can be marked by adding an irregular bounding box. For example, when distinguishing the second region from other regions, the program directly concatenates the pixels at the edges of the second region, ultimately adding an irregular bounding box to the second region.

[0083] In summary, please refer to the following: Figure 3 Step 240 is... Figure 3 The right half of the image. After the second image 321 undergoes DCT transformation and filtering, a second DCT matrix is ​​obtained. By inputting the second DCT matrix into the compressor and decompressor, the second region can be marked. Figure 3 The rectangle 322 in the image is the marker added to the second region. A second virtual object (a soldier) exists within this second region.

[0084] Figure 3 The right half of the image also shows a recorder 323, which is used to record the type and position of virtual objects within the second image. Schematably, three circles with filled content in the recorder 323 represent three virtual objects, "a cannon and two soldiers," within the second image. Circles with different filled content represent different virtual objects. Furthermore, the positions of the three filled circles in the recorder 323 correspond to the positions of the three virtual objects in the second image; the circle on the left represents the virtual object on the left side of the second image, and the circle on the right represents the virtual object on the right side. This pattern continues, meaning that when multiple virtual objects exist in the second image, the recorder 323 records them sequentially. Therefore, the recorder 323 can record the virtual objects within the second image, and by combining the records of adjacent images, the movement trajectory of any virtual object can be obtained.

[0085] Step 250: Based on the image features of the first region and the second region, determine that the first virtual object and the second virtual object are the same virtual object.

[0086] As shown above, the first virtual object corresponds to the first region of the first image, and the second virtual object corresponds to the second region of the second image. Based on the image features of the first and second regions, it can be determined that the first virtual object and the second virtual object are the same virtual object.

[0087] In one embodiment, the first virtual object and the second virtual object are determined to be the same virtual object by at least one of the following two methods.

[0088] The first possible method of determination: Based on the correspondence between the first location information and the second location information, determine that the first virtual object and the second virtual object are the same virtual object; wherein, the first location information is the location information of the first region in the first image, and the second location information is the location information of the second region in the second image.

[0089] In one embodiment, if the difference between the coordinates of the first region in the first image and the coordinates of the second region in the second image is less than a threshold, then the first virtual object and the second virtual object are considered to be the same virtual object.

[0090] In another embodiment, multiple regions are detected in the second image, each region corresponding to a virtual object. The virtual object corresponding to the second region whose position information is closest to that of the first region is considered to be the same virtual object as the virtual object corresponding to the first region.

[0091] For example, if the coordinates of the first region in the first image are detected as (20, 25), and the coordinates of another region in the first image are detected as (60, 60), and the coordinates of the second region in the second image are detected as (22, 26), and the coordinates of another region in the second image are detected as (61, 62), then it is clear that the coordinates of the first region in the first image and the coordinates of the second region in the second image are relatively close, so the first virtual object and the second virtual object are considered to be the same virtual object.

[0092] The second possible method of determination: Based on the similarity between the first region and the second region, determine that the first virtual object and the second virtual object are the same virtual object.

[0093] In one embodiment, the similarity between the first region and the second region is calculated. If the similarity between the first region and the second region is less than a threshold, then the first virtual object and the second virtual object are determined to be the same virtual object.

[0094] Optionally, the similarity between the first region and the second region can be calculated based on at least one of resolution, region size, color, hue, saturation, and brightness. For example, if the first region and the second region are the same size, then the first virtual object and the second virtual object are determined to be the same virtual object. Or, for example, if the first region and the second region have the same color types and number of colors, then the first virtual object and the second virtual object are determined to be the same virtual object.

[0095] In summary, a method for identifying virtual objects is provided by marking a first region corresponding to a first virtual object based on the frequency domain information of the first image, and marking a second region corresponding to a second virtual object based on the frequency domain information of the second image. Then, based on the image features of the first and second regions, it is determined that the first and second virtual objects are the same virtual object. This method not only supports the identification of virtual objects in image frames, but also supports the one-to-one correspondence between virtual objects in two image frames and the acquisition of the movement trajectory of the virtual object.

[0096] Furthermore, by performing discrete cosine transform and filtering operations, a first DCT matrix storing information about the first virtual object is obtained. The virtual object is then labeled based on the first DCT matrix, and the image features of the first virtual object in the first image are extracted. Based on the image features of the first virtual object, attention weights are assigned to the first image through an attention mechanism, which distinguishes the first region from other regions. Finally, a label is added to the first region, providing a specific implementation method that improves the accuracy and efficiency of labeling the first virtual object.

[0097] Furthermore, by performing discrete cosine transform and filtering operations, a second DCT matrix storing information about the second virtual object is obtained. The virtual object is then labeled based on the second DCT matrix, and the image features of the second virtual object in the second image are extracted. Based on the image features of the second virtual object, attention weights are assigned to the second image through an attention mechanism, thus distinguishing the second region from other regions. Finally, a label is added to the second region, improving the accuracy and efficiency of labeling the second virtual object.

[0098] based on Figure 2 In the optional embodiment shown, step 220 may involve: assigning a first attention weight to the matrix block indicating the first region in the first DCT matrix using an attention mechanism, thereby obtaining a weighted first DCT matrix. Step 240 may involve: assigning a second attention weight to the matrix block indicating the second region in the second DCT matrix using an attention mechanism, thereby obtaining a weighted second DCT matrix.

[0099] In one embodiment, a compressor is used to assign attention weights to the DCT matrix.

[0100] Reference Figure 4 The compressor 400 includes M compression layers. Each compression layer contains an attention network and a fully connected network. The attention network divides the DCT matrix into multiple matrix blocks and assigns attention weights to the matrix blocks that indicate regions corresponding to virtual objects. The fully connected network connects the weighted matrix blocks according to the positional relationships of the image's pixel matrix.

[0101] Optionally, each compression layer is used to assign attention weights to a region corresponding to a virtual object, and M compression layers can assign attention weights to a total of M regions. Optionally, each compression layer is used to assign attention weights to all regions corresponding to virtual objects, and M compression layers are used to repeatedly assign attention weights.

[0102] For example, the following formula (5) shows the compression function of the compression layer.

[0103] f = exp(x + sublayer(x)); (5)

[0104] Where x is the input DCT matrix, and sublayer(x) is the output matrix of the previous compression layer. For the first compression layer, sublayer(x) is 0.

[0105] Compression processing for the first image:

[0106] 1-1, Input the first DCT matrix into the first compression layer to obtain the output matrix of the first compression layer;

[0107] 1-2, Input the first DCT matrix and the output matrix of the (m-1)th compression layer into the mth compression layer to obtain the output matrix of the mth compression layer; the initial value of m is 2;

[0108] 1-3. If m is less than M, update m to m+1 and re-execute the step of inputting the first DCT matrix and the output matrix of the (m-1)th compression layer into the mth compression layer to obtain the output matrix of the mth compression layer (i.e., step 1-2); M is the preset number of compression layers;

[0109] 1-4. When m equals M, the output matrix of the Mth compressed layer is determined as the weighted first DCT matrix.

[0110] Compression processing for the second image:

[0111] 2-1, Input the second DCT matrix into the first compression layer to obtain the output matrix of the first compression layer;

[0112] 2-2, Input the second DCT matrix and the output matrix of the (m-1)th compression layer into the mth compression layer to obtain the output matrix of the mth compression layer; the initial value of m is 2;

[0113] 2-3. If m is less than M, update m to m+1 and re-execute the step of inputting the second DCT matrix and the output matrix of the (m-1)th compression layer into the mth compression layer to obtain the output matrix of the mth compression layer (i.e., step 2-2); M is the preset number of compression layers;

[0114] 2-4. When m equals M, the output matrix of the Mth compression layer is determined as the weighted second DCT matrix.

[0115] In summary, the above method provides a specific structure for the compressor and further offers a method for assigning attention weights to the first image based on the image features of the first virtual object through an attention mechanism, thereby improving the accuracy and efficiency of labeling the first virtual object.

[0116] Furthermore, the above method provides a specific structure for the compressor and further offers a method for assigning attention weights to the second image based on the image features of the second virtual object through an attention mechanism, thereby improving the accuracy and efficiency of labeling the second virtual object.

[0117] based on Figure 2 In the optional embodiment shown, step 220 may involve marking a first region within the first image based on the weighted first DCT matrix and the pixel matrix of the first image. Step 240 may involve marking a second region within the second image based on the weighted second DCT matrix and the pixel matrix of the second image.

[0118] In one embodiment, a decompressor adds labels to the image based on the weighted DCT matrix.

[0119] Reference Figure 5 The decompressor consists of N decompression layers. Each decompression layer contains an attention network and an interpolation network. The attention network normalizes the weighted DCT matrix and multiplies the normalized result by the image's pixel matrix. The interpolation network enlarges the image; optionally, the interpolation network uses linear interpolation and / or mean interpolation.

[0120] Optionally, each decompression layer normalizes a matrix block corresponding to a region containing a virtual object, and multiplies the normalization result by the pixel block of that region in the image's pixel matrix. N decompression layers can be used to stack the normalization results for a total of N regions. Alternatively, each decompression layer normalizes a matrix block for all regions corresponding to virtual objects, and multiplies the normalization result by the pixel block of all regions in the image's pixel matrix. N decompression layers are used to repeatedly stack the normalization results.

[0121] For example, the following formula (6) shows the decompression function of the decompression layer.

[0122] f = softmax(F / exp)V; (6)

[0123] Where F is the weighted DCT matrix or the output matrix of the previous decompression layer, exp is the exponent e, and V is the pixel matrix of the image.

[0124] Decompression processing for the first image:

[0125] 1-1' Input the weighted first DCT matrix and the pixel matrix of the first image into the first decompression layer to obtain the output matrix of the first decompression layer;

[0126] 1-2' Input the pixel matrix of the first image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer; the initial value of n is 2;

[0127] 1-3', If n is less than N, update n to n+1, and re-execute the step of inputting the pixel matrix of the first image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer (i.e., step 1-2'); N is the preset number of decompression layers;

[0128] 1-4', when n equals N, mark the second region on the first image based on the output matrix of the Nth decompression layer;

[0129] Decompression processing for the second image:

[0130] 2-1', Input the weighted second DCT matrix and the pixel matrix of the i-th frame image into the first decompression layer to obtain the output matrix of the first decompression layer;

[0131] 2-2' Input the pixel matrix of the second image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer; the initial value of n is 2;

[0132] 2-3', If n is less than N, update n to n+1, and re-execute the step of inputting the pixel matrix of the second image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer; N is the preset number of decompression layers;

[0133] 2-4', when n equals N, mark the second region on the second image based on the output matrix of the Nth decompression layer.

[0134] In summary, the above method provides a specific structure for the decompression unit. Based on the image features of the first virtual object after attention weighting, it distinguishes the first region from other regions and finally adds a label to the first region, thereby improving the accuracy and efficiency of labeling the first virtual object.

[0135] Furthermore, based on the image features of the second virtual object after attention weighting, the second region is distinguished from other regions, and finally, a label is added to the second region, which improves the accuracy and efficiency of labeling the second virtual object.

[0136] In the above Figure 2The illustrated method embodiment provides a method for identifying virtual objects, which supports a one-to-one correspondence between virtual objects in two image frames. Further, a method for identifying the generation and disappearance of each virtual object will be provided.

[0137] based on Figure 2 In the alternative embodiments shown, Figure 6 A flowchart of a virtual object identification method provided in an exemplary embodiment of this application is shown, the method being... Figure 2 In the method shown, the first image is redefined as a keyframe image, and the second image is redefined as the (i-1)th frame image. This method is derived from... Figure 1 The terminal or server shown is used as an example for illustration. Figure 6 The methods shown include:

[0138] Step 610: Based on the frequency domain information of the (i-1)th frame image, mark the third and fourth regions within the (i-1)th frame image;

[0139] In the above Figure 2 In step 240, a method for marking the second region in the second image based on the frequency domain information of the second image is given. Using a similar method, the third and fourth regions can be marked in the (i-1)th frame image (the second image), and virtual objects are corresponding to both the third and fourth regions.

[0140] Reference Figure 7 It shows the third region 701 and the fourth region 702 within the i-1th frame image.

[0141] Optionally, step 610 in Figure 2 The method embodiment shown may be executed before or after step 240, or step 610 may be executed simultaneously with step 240.

[0142] Step 620: Based on the image features of the keyframe image and the (i-1)th frame image, determine that the virtual object in the third region is a newly added virtual object in the (i-1)th frame image;

[0143] A keyframe image refers to an image used to identify newly added virtual objects in a video stream based on a virtual environment. Optionally, determining that a virtual object in a third region is a newly added virtual object in the (i-1)th frame image based on the image features of the keyframe image and the (i-1)th frame image can include at least one of the following methods:

[0144] First, based on the location information of the region marked by the keyframe image and the (i-1)th frame image, the newly added virtual object is determined.

[0145] For example, if the coordinates of all regions corresponding to virtual objects in the keyframe image are different from the coordinates of the third region in the (i-1)th frame, then it is determined that there is no region corresponding to the third region in the keyframe image, that is, the virtual object in the third region is determined to be a newly added virtual object.

[0146] Second, based on the similarity between the regions marked in the keyframe image and the (i-1)th frame image, the newly added virtual objects are determined.

[0147] Optionally, the similarity between all regions corresponding to virtual objects within the keyframe image and the third region can be calculated based on at least one of resolution, region size, color, hue, saturation, and brightness.

[0148] For example, if the size of all regions corresponding to virtual objects in the keyframe image is inconsistent with that of the third region, then the virtual objects in the third region are determined to be newly added virtual objects. Similarly, if the color type and number of colors in all regions corresponding to virtual objects in the keyframe image are inconsistent with those in the third region, then the virtual objects in the third region are determined to be newly added virtual objects.

[0149] Optionally, step 620 in Figure 2 The method embodiment shown may be executed before or after step 250, or step 620 may be executed simultaneously with step 250.

[0150] Step 630: Obtain the i-th frame image from the video stream based on the virtual environment; based on the image features of the (i-1)-th frame image and the i-th frame image, determine the virtual object in the fourth region as the virtual object reduced from the i-th frame image.

[0151] The i-th frame refers to the frame following the (i-1)-th frame in the video stream based on the virtual environment. Optionally, determining the virtual objects in the fourth region as the virtual objects reduced from the i-th frame image based on the image features of the (i-1)-th frame and the i-th frame can include at least one of the following methods:

[0152] First, based on the location information of the region marked in the (i-1)th frame image and the i-th frame image, the number of virtual objects to be reduced is determined.

[0153] For example, if the coordinates of all regions corresponding to virtual objects in the i-th frame image differ from the coordinates of the fourth region in the (i-1)-th frame by a threshold, then it is determined that there is no region corresponding to the fourth region in the i-th frame image, that is, the virtual objects in the fourth region are determined to be reduced virtual objects.

[0154] Second, based on the similarity between the regions marked in the (i-1)th frame image and the i-th frame image, the number of virtual objects to be reduced is determined.

[0155] Optionally, the similarity between all regions in the i-th frame image that correspond to virtual objects and the fourth region can be calculated based on at least one of resolution, region size, color, hue, saturation, and brightness.

[0156] For example, if the size of all regions corresponding to virtual objects in the i-th frame is different from that of the fourth region, then the virtual objects in the fourth region are determined to be reduced virtual objects. Similarly, if the color type and number of colors in all regions corresponding to virtual objects in the i-th frame are different from those in the fourth region, then the virtual objects in the fourth region are determined to be reduced virtual objects.

[0157] Optionally, step 630 in Figure 2 Step 250 of the method embodiment shown is executed afterward.

[0158] In summary, based on the image features of the (i-1)th frame and the keyframe image, newly added virtual objects are identified, and based on the image features of the i-th frame and the (i-1)th frame image, reduced virtual objects are identified. This provides a method for determining the generation and disappearance of virtual objects, and further determines the movement trajectory of the virtual objects.

[0159] based on Figure 2 or Figure 6 In the optional embodiment shown, there may also be a step S1 related to selecting keyframe images, based on Figure 2 In the optional embodiment shown, step S1 is located after step 250; based on Figure 2 In the alternative embodiment shown, step S1 is located before or after any of steps 610, 620 and 630.

[0160] S1: If the difference between the area of ​​the first region of the keyframe image and the area of ​​the second region of the (i-1)th frame image satisfies the threshold, the (i-1)th frame image is identified as a new keyframe image.

[0161] Optionally, if the first virtual object detected in the first region of the keyframe image and the second virtual object detected in the second region of the (i-1)th frame image are the same virtual object, and the difference between the area of ​​the first region and the area of ​​the second region (i.e., the area of ​​the first virtual object and the area of ​​the second virtual object) reaches a threshold, then it is considered that there is a large change between the keyframe image and the (i-1)th frame image, and the (i-1)th frame image is confirmed as a new keyframe image.

[0162] Optionally, if the first virtual object is a complete virtual object, the second virtual object is the same virtual object as the first virtual object, and the missing proportion of the second virtual object compared to the complete virtual object reaches a threshold, the i-1th frame image is identified as a new keyframe image.

[0163] In summary, when the degree of disappearance of the second virtual object in the (i-1)th frame image compared to the first virtual object in the keyframe image reaches a threshold, the (i-1)th frame image is determined as a new keyframe image, wherein the second virtual object and the first virtual object are the same virtual object, providing a method for determining keyframes.

[0164] In an optional embodiment, the method for identifying virtual objects is mainly divided into three stages: target initialization discovery, target incremental identification, and target ablation identification.

[0165] In the target initialization and discovery phase, dynamic targets are primarily identified by fusing DCT and decompression techniques, and each target is marked with a different color. In the target incremental recognition phase, new incremental targets are identified using the multi-target location and feature information from the previous frame (frame i-1), again employing DCT and decompression techniques. During this process, newly discovered targets are marked with different colors. In the target ablation recognition phase, reduced targets are identified using the multi-target location and feature information from the next frame (frame i), and these targets are discarded.

[0166] DCT: The entire image is transformed in the frequency domain through discrete cosine transformation, and then high-frequency features are filtered out through a custom high-frequency filter. The image with these features mainly contains rich information about the game characters.

[0167] Decompression technology: During compression, an attention mechanism is incorporated to break down the entire image's DCT matrix into independent compressed blocks. During decompression, image processing techniques are used to reconstruct the compressed data into images corresponding to individual image targets, thus supporting automatic target labeling.

[0168] In the compression process, compression is performed using a compressor. The compressor provides a fixed-dimensional vector, and the attention mechanism is used to adjust or increase the weight of core information. The compressor consists of six identical compression layers stacked together, with each compression layer having two sub-layers. The first sub-layer is a multi-head self-attention mechanism, and the second sub-layer is a simple positionally fully connected feedforward network. A residual connection is used between the two sub-layers, followed by layer normalization, meaning that the compression function of each compression layer is exp(x + sublayer(x)). Furthermore, the output and input of each sub-layer have the same dimension. Therefore, by passing the results of DCT and filtering through the compressor network, the attention information is compressed and transformed into a compressed vector matrix.

[0169] The decompression process aims to restore image information. Since low-frequency and ultra-high-frequency information has been filtered out, leaving only attention-related information, the decompressor only needs to restore the attention-related information.

[0170] The decompressor also consists of six identical decompression layers. Each decompression layer in the decompressor is formed by inserting a multi-target attention mechanism and a weighted mean filter into the compression layer of the compressor. The data output from the compressor and the pixel matrix of the original image serve as the input to the decompressor. After passing through a multi-target attention mechanism and a mean filter layer, the output of layer MA-1 (the first layer of the multi-target attention mechanism) serves as the input to the next layer MA-2, and the key and value of layer MA-2. The output of layer MA-2 is input to a feedforward layer (FF). After multiple operations and a linear transformation, the final result is a labeled character tag, which is an adaptive box automatically added based on the attention of the recovered image. The multi-target attention mechanism function is MA = softmax(F / exp)V, where MA is the multi-target attention matrix, F is the output from the compressor, V is the original image, exp is the exponent e, and softmax is the softmax function.

[0171] Figure 8 This invention illustrates a structural block diagram of a virtual object recognition device provided in an exemplary embodiment of the present application. The device includes:

[0172] Acquisition module 801 is used to acquire a first image from a video stream based on a virtual environment;

[0173] The marking module 802 is used to mark a first region in the first image based on the frequency domain information of the first image, and the first region corresponds to a first virtual object;

[0174] The acquisition module 801 is also used to acquire a second image from a video stream based on a virtual environment;

[0175] The marking module 802 is also used to mark a second region in the second image based on the frequency domain information of the second image, and the second region corresponds to a second virtual object;

[0176] The determination module 803 is used to determine that the first virtual object and the second virtual object are the same virtual object based on the image features of the first region and the second region.

[0177] In an optional embodiment, the marking module 802 is further configured to obtain the frequency domain information of the first image by performing a discrete cosine transform (DCT); input the frequency domain information of the first image into a filter for recognizing virtual objects to obtain a first DCT matrix, the first DCT matrix storing information of the first virtual object; and mark a first region in the first image according to the first DCT matrix.

[0178] In an optional embodiment, the marking module 802 is further configured to obtain the frequency domain information of the second image by passing the second image through DCT; input the frequency domain information of the second image into a filter for recognizing virtual objects to obtain a second DCT matrix, the second DCT matrix storing information of the second virtual object; and mark a second region in the second image according to the second DCT matrix.

[0179] In an optional embodiment, the marking module 802 is further configured to assign a first attention weight to the matrix block indicating the first region in the first DCT matrix using an attention mechanism to obtain a weighted first DCT matrix; and to mark the first region in the first image based on the weighted first DCT matrix and the pixel matrix of the first image.

[0180] In an optional embodiment, the marking module 802 is further configured to assign a second attention weight to the matrix block indicating the second region in the second DCT matrix using an attention mechanism to obtain a weighted second DCT matrix; and to mark the second region in the second image based on the weighted second DCT matrix and the pixel matrix of the second image.

[0181] In an optional embodiment, the labeling module 802 is further configured to input the first DCT matrix into the first compression layer to obtain the output matrix of the first compression layer; input the first DCT matrix and the output matrix of the (m-1)th compression layer into the m compression layer to obtain the output matrix of the m compression layer; the initial value of m is 2; if m is less than M, m is updated to m+1, and the step of inputting the first DCT matrix and the output matrix of the (m-1)th compression layer into the m compression layer to obtain the output matrix of the m compression layer is executed again; M is the preset number of compression layers; if m is equal to M, the output matrix of the M compression layer is determined as the weighted first DCT matrix; wherein, the compression layer includes an attention network and a fully connected network.

[0182] In an optional embodiment, the labeling module 802 is further configured to input the second DCT matrix into the first compression layer to obtain the output matrix of the first compression layer; input the second DCT matrix and the output matrix of the (m-1)th compression layer into the m compression layer to obtain the output matrix of the m compression layer; the initial value of m is 2; if m is less than M, m is updated to m+1, and the step of inputting the second DCT matrix and the output matrix of the (m-1)th compression layer into the m compression layer to obtain the output matrix of the m compression layer is executed again; M is the preset number of compression layers; if m is equal to M, the output matrix of the M compression layer is determined as the weighted second DCT matrix; wherein, the compression layer includes an attention network and a fully connected network.

[0183] In an optional embodiment, the marking module 802 is further configured to input the weighted first DCT matrix and the pixel matrix of the first image into the first decompression layer to obtain the output matrix of the first decompression layer; input the pixel matrix of the first image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer; the initial value of n is 2; if n is less than N, n is updated to n+1, and the step of inputting the pixel matrix of the first image and the output matrix of the (n-1)th decompression layer into the nth decompression layer is executed again to obtain the output matrix of the nth decompression layer; N is the preset number of decompression layers; if n is equal to N, the second region is marked on the first image based on the output matrix of the Nth decompression layer; wherein, the decompression layer includes an attention network and an interpolation network.

[0184] In an optional embodiment, the marking module 802 is further configured to input the weighted second DCT matrix and the pixel matrix of the i-th frame image into the first decompression layer to obtain the output matrix of the first decompression layer; input the pixel matrix of the second image and the output matrix of the (n-1)-th decompression layer into the n-th decompression layer to obtain the output matrix of the n-th decompression layer; the initial value of n is 2; if n is less than N, n is updated to n+1, and the step of inputting the pixel matrix of the second image and the output matrix of the (n-1)-th decompression layer into the n-th decompression layer is executed again to obtain the output matrix of the n-th decompression layer; N is the preset number of decompression layers; if n is equal to N, the second region is marked on the second image based on the output matrix of the N-th decompression layer; wherein, the decompression layer includes an attention network and an interpolation network.

[0185] In an optional embodiment, the determining module 803 is further configured to determine that the first virtual object and the second virtual object are the same virtual object based on the relationship between the first location information and the second location information; the first location information is the location information of the first region in the first image, and the second location information is the location information of the second region in the second image.

[0186] In an optional embodiment, the determining module 803 determines that the first virtual object and the second virtual object are the same virtual object based on the similarity between the first region and the second region.

[0187] In an optional embodiment, the first image is a keyframe image, and the second image is the (i-1)th frame image; the marking module 802 is further configured to mark a third region in the (i-1)th frame image based on the frequency domain information of the (i-1)th frame image; the determining module 803 is configured to determine that the third virtual object is a newly added virtual object in the (i-1)th frame image when the keyframe image has not been marked with a region corresponding to the third region.

[0188] In an optional embodiment, the determining module 803 is further configured to identify the (i-1)th frame image as a new keyframe image if the difference between the area of ​​the first region of the keyframe image and the area of ​​the second region of the (i-1)th frame image satisfies a threshold.

[0189] In an optional embodiment, the marking module 802 is further configured to mark a fourth region in the (i-1)th frame image based on the frequency domain information of the (i-1)th frame image; the acquisition module 801 is further configured to acquire the i-th frame image in the video stream based on the virtual environment; and the determination module 803 is further configured to determine that the fourth virtual object is a virtual object reduced from the i-th frame image if the i-th frame image does not have a region corresponding to the fourth region marked.

[0190] In summary, a device for identifying virtual objects is provided by marking a first region corresponding to a first virtual object based on the frequency domain information of a first image, and marking a second region corresponding to a second virtual object based on the frequency domain information of a second image. Then, based on the image features of the first and second regions, it is determined that the first and second virtual objects are the same virtual object. This device not only supports the identification of virtual objects in image frames, but also supports the one-to-one correspondence between virtual objects in two image frames and the acquisition of the movement trajectory of the virtual objects.

[0191] Figure 9 This illustration shows a structural block diagram of a computer device 900 provided in an exemplary embodiment of this application. The computer device 900 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computer device 900 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0192] Typically, computer device 900 includes a processor 901 and a memory 902.

[0193] Processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0194] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one instruction, which is executed by the processor 901 to implement the virtual object identification method provided in the method embodiments of this application.

[0195] In some embodiments, the computer device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. For example, the peripheral device may include at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0196] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0197] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 904 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 9G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0198] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, disposed on the front panel of computer device 900; in other embodiments, there may be at least two display screens 905, disposed on different surfaces of computer device 900 or in a folded design; in other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of computer device 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0199] The camera assembly 906 is used to acquire images or videos. Optionally, the camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0200] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 901 for processing, or input to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the computer device 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0201] Power supply 908 is used to supply power to the various components in computer device 900. Power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0202] In some embodiments, the computer device 900 further includes one or more sensors 909. The one or more sensors 909 include, but are not limited to, an accelerometer 910, a gyroscope 911, a pressure sensor 912, an optical sensor 913, and a proximity sensor 914.

[0203] Accelerometer 910 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by computer device 900. For example, accelerometer 910 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 901 can control display screen 905 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 910. Accelerometer 910 can also be used for games or for acquiring user motion data.

[0204] The gyroscope sensor 911 can detect the orientation and rotation angle of the computer device 900. The gyroscope sensor 911, in conjunction with the accelerometer sensor 910, can collect 3D motion data from the user on the computer device 900. Based on the data collected by the gyroscope sensor 911, the processor 901 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0205] The pressure sensor 912 can be disposed on the side bezel of the computer device 900 and / or on the lower layer of the display screen 905. When the pressure sensor 912 is disposed on the side bezel of the computer device 900, it can detect the user's grip signal on the computer device 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 912. When the pressure sensor 912 is disposed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0206] An optical sensor 913 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity collected by the optical sensor 913. For example, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 913.

[0207] A proximity sensor 914, also known as a distance sensor, is typically located on the front panel of a computer device 900. The proximity sensor 914 is used to detect the distance between the user and the front of the computer device 900. In one embodiment, when the proximity sensor 914 detects that the distance between the user and the front of the computer device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 914 detects that the distance between the user and the front of the computer device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from a screen-off state to a screen-on state.

[0208] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the computer device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0209] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the virtual object identification method provided in the above method embodiments.

[0210] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the virtual object identification method provided in the above-described method embodiments.

[0211] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0212] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0213] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for identifying virtual objects, characterized in that, The method includes: A first image is acquired from a video stream based on a virtual environment; the first image is processed by Discrete Cosine Transform (DCT) to obtain its frequency domain information; the frequency domain information of the first image is input into a filter used to identify virtual objects to obtain a first DCT matrix, which stores information about the first virtual object; an attention mechanism is used to assign a first attention weight to the matrix block in the first DCT matrix that indicates the first region, resulting in a weighted first DCT matrix; based on the weighted first DCT matrix and the pixel matrix of the first image, the first region is marked within the first image, and the first region corresponds to the first virtual object. A second image is acquired from the video stream based on the virtual environment; the second image is processed by DCT to obtain its frequency domain information; the frequency domain information of the second image is input into the filter used to identify virtual objects to obtain a second DCT matrix, which stores information about the second virtual object; the attention mechanism is used to assign a second attention weight to the matrix block indicating the second region in the second DCT matrix to obtain a weighted second DCT matrix; based on the weighted second DCT matrix and the pixel matrix of the second image, the second region is marked in the second image, and the second region corresponds to the second virtual object. Based on the image features of the first region and the second region, it is determined that the first virtual object and the second virtual object are the same virtual object; Specifically, a compressor is used to assign the first attention weight to the first DCT matrix and the second attention weight to the second DCT matrix; a decompressor is used to label the first region of the first image based on the weighted first DCT matrix and to label the second region of the second image based on the weighted second DCT matrix.

2. The method according to claim 1, characterized in that, The compressor includes m compression layers. The step of employing an attention mechanism to assign first attention weights to the matrix blocks indicating the first region in the first DCT matrix, resulting in a weighted first DCT matrix, includes: The first DCT matrix is ​​input into the first compression layer to obtain the output matrix of the first compression layer; The first DCT matrix and the output matrix of the (m-1)th compression layer are input into the mth compression layer to obtain the output matrix of the mth compression layer; the initial value of m is 2. If m is less than M, update m to m+1, and re-execute the step of inputting the first DCT matrix and the output matrix of the (m-1)th compression layer into the mth compression layer to obtain the output matrix of the mth compression layer; M is the preset number of compression layers; When m equals M, the output matrix of the Mth compression layer is determined as the weighted first DCT matrix; The compression layer includes an attention network and a fully connected network, where m and M are both positive integers.

3. The method according to claim 1, characterized in that, The compressor includes m compression layers. The step of employing the attention mechanism to assign second attention weights to the matrix blocks indicating the second region in the second DCT matrix, resulting in a weighted second DCT matrix, includes: The second DCT matrix is ​​input into the first compression layer to obtain the output matrix of the first compression layer; The second DCT matrix and the output matrix of the (m-1)th compression layer are input into the mth compression layer to obtain the output matrix of the mth compression layer; the initial value of m is 2. If m is less than M, update m to m+1, and re-execute the step of inputting the second DCT matrix and the output matrix of the (m-1)th compression layer into the mth compression layer to obtain the output matrix of the mth compression layer; M is the preset number of compression layers; When m equals M, the output matrix of the Mth compression layer is determined as the weighted second DCT matrix; The compression layer includes an attention network and a fully connected network, where m and M are both positive integers.

4. The method according to claim 1, characterized in that, The decompressor includes n decompression layers, and the step of marking the first region on the first image based on the weighted first DCT matrix and the pixel matrix of the first image includes: The weighted first DCT matrix and the pixel matrix of the first image are input into the first decompression layer to obtain the output matrix of the first decompression layer; The pixel matrix of the first image and the output matrix of the (n-1)th decompression layer are input into the nth decompression layer to obtain the output matrix of the nth decompression layer; the initial value of n is 2; If n is less than N, update n to n+1 and re-execute the step of inputting the pixel matrix of the first image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer; N is the preset number of decompression layers; When n equals N, the first region is marked on the first image based on the output matrix of the Nth decompression layer; The decompression layer includes an attention network and an interpolation network, where n and N are both positive integers.

5. The method according to claim 1, characterized in that, The decompressor includes n decompression layers. Marking the second region on the second image based on the weighted second DCT matrix and the pixel matrix of the second image includes: The weighted second DCT matrix and the pixel matrix of the second image are input into the first decompression layer to obtain the output matrix of the first decompression layer; The pixel matrix of the second image and the output matrix of the (n-1)th decompression layer are input into the nth decompression layer to obtain the output matrix of the nth decompression layer; the initial value of n is 2; If n is less than N, update n to n+1 and re-execute the step of inputting the pixel matrix of the second image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer; N is the preset number of decompression layers; When n equals N, the second region is marked on the second image based on the output matrix of the Nth decompression layer; The decompression layer includes an attention network and an interpolation network, where n and N are both positive integers.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining that the first virtual object and the second virtual object are the same virtual object based on the image features of the first region and the second region includes: Based on the relationship between the first location information and the second location information, it is determined that the first virtual object and the second virtual object are the same virtual object; the first location information is the location information of the first region in the first image, and the second location information is the location information of the second region in the second image; And / or, Based on the similarity between the first region and the second region, it is determined that the first virtual object and the second virtual object are the same virtual object.

7. The method according to any one of claims 1 to 5, characterized in that, The first image is a keyframe image, and the second image is the (i-1)th frame image; the method further includes: Based on the frequency domain information of the (i-1)th frame image, the third region corresponding to the third virtual object is marked in the (i-1)th frame image; If the keyframe image does not have a region corresponding to the third region marked, the third virtual object is determined to be a newly added virtual object in the (i-1)th frame image.

8. The method according to claim 7, characterized in that, The method further includes: If the difference between the area of ​​the first region of the keyframe image and the area of ​​the second region of the (i-1)th frame image satisfies a threshold, the (i-1)th frame image is identified as a new keyframe image.

9. The method according to any one of claims 1 to 5, characterized in that, The second image is the (i-1)th frame image, and the method further includes: Based on the frequency domain information of the (i-1)th frame image, the fourth region corresponding to the fourth virtual object is marked in the (i-1)th frame image; Obtain the i-th frame image from the video stream based on the virtual environment; If the i-th frame image does not have a region corresponding to the fourth region marked, the fourth virtual object is determined to be a virtual object reduced from the i-th frame image.

10. A device for recognizing virtual objects, characterized in that, The device includes: An acquisition module is used to acquire a first image from a video stream based on a virtual environment; A labeling module is used to obtain the frequency domain information of the first image by performing a Discrete Cosine Transform (DCT); input the frequency domain information of the first image into a filter for recognizing virtual objects to obtain a first DCT matrix, which stores information about the first virtual object; assign a first attention weight to the matrix block indicating the first region in the first DCT matrix using an attention mechanism to obtain a weighted first DCT matrix; and label the first region in the first image according to the weighted first DCT matrix and the pixel matrix of the first image, with the first region corresponding to the first virtual object. The acquisition module is further configured to acquire a second image from the video stream based on the virtual environment; The marking module is further configured to: obtain frequency domain information of the second image by passing the DCT through the second image; input the frequency domain information of the second image into the filter used to identify virtual objects to obtain a second DCT matrix, the second DCT matrix storing information of the second virtual object; assign a second attention weight to the matrix block indicating the second region in the second DCT matrix using the attention mechanism to obtain a weighted second DCT matrix; and mark the second region in the second image according to the weighted second DCT matrix and the pixel matrix of the second image, the second region corresponding to the second virtual object. The determining module is used to determine, based on the image features of the first region and the second region, that the first virtual object and the second virtual object are the same virtual object; Specifically, a compressor is used to assign the first attention weight to the first DCT matrix and the second attention weight to the second DCT matrix; a decompressor is used to label the first region of the first image based on the weighted first DCT matrix and to label the second region of the second image based on the weighted second DCT matrix.

11. The apparatus according to claim 10, characterized in that, The compressor comprises m compression layers; The labeling module is used to input the first DCT matrix into the first compression layer to obtain the output matrix of the first compression layer; input the first DCT matrix and the output matrix of the (m-1)th compression layer into the m-th compression layer to obtain the output matrix of the m-th compression layer; the initial value of m is 2; if m is less than M, m is updated to m+1, and the step of inputting the first DCT matrix and the output matrix of the (m-1)th compression layer into the m-th compression layer to obtain the output matrix of the m-th compression layer is executed again; M is a preset number of compression layers; if m is equal to M, the output matrix of the M-th compression layer is determined as the weighted first DCT matrix; wherein, the compression layer includes an attention network and a fully connected network.

12. The apparatus according to claim 10, characterized in that, The compressor comprises m compression layers; The labeling module is used to input the second DCT matrix into the first compression layer to obtain the output matrix of the first compression layer; input the second DCT matrix and the output matrix of the (m-1)th compression layer into the m-th compression layer to obtain the output matrix of the m-th compression layer; the initial value of m is 2; if m is less than M, m is updated to m+1, and the step of inputting the second DCT matrix and the output matrix of the (m-1)th compression layer into the m-th compression layer to obtain the output matrix of the m-th compression layer is executed again; M is a preset number of compression layers; if m is equal to M, the output matrix of the M-th compression layer is determined as the weighted second DCT matrix; wherein, the compression layer includes an attention network and a fully connected network.

13. The apparatus according to claim 10, characterized in that, The decompressor includes n decompression layers, and the marking module is further used to input the weighted first DCT matrix and the pixel matrix of the first image into the first decompression layer to obtain the output matrix of the first decompression layer; The pixel matrix of the first image and the output matrix of the (n-1)th decompression layer are input into the nth decompression layer to obtain the output matrix of the nth decompression layer; the initial value of n is 2. If n is less than N, update n to n+1 and re-execute the step of inputting the pixel matrix of the first image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer; N is the preset number of decompression layers; When n equals N, the first region is marked on the first image based on the output matrix of the Nth decompression layer; wherein the decompression layer includes an attention network and an interpolation network.

14. The apparatus according to claim 10, characterized in that, The decompressor includes n decompression layers. The marking module is further used to input the weighted second DCT matrix and the pixel matrix of the second image into the first decompression layer to obtain the output matrix of the first decompression layer. The pixel matrix of the second image and the output matrix of the (n-1)th decompression layer are input into the nth decompression layer to obtain the output matrix of the nth decompression layer; the initial value of n is 2. If n is less than N, update n to n+1 and re-execute the step of inputting the pixel matrix of the second image and the output matrix of the (n-1)th decompression layer into the nth decompression layer to obtain the output matrix of the nth decompression layer; N is the preset number of decompression layers; When n equals N, the second region is marked on the second image based on the output matrix of the Nth decompression layer; wherein the decompression layer includes an attention network and an interpolation network.

15. The apparatus according to any one of claims 10 to 14, characterized in that, The determining module is further configured to determine, based on the relationship between the first location information and the second location information, that the first virtual object and the second virtual object are the same virtual object; the first location information is the location information of the first region in the first image, and the second location information is the location information of the second region in the second image; or, The determining module is used to determine that the first virtual object and the second virtual object are the same virtual object based on the similarity between the first region and the second region.

16. The apparatus according to any one of claims 10 to 14, characterized in that, The first image is a keyframe image, and the second image is the (i-1)th frame image; The marking module is used to mark the third region corresponding to the third virtual object in the (i-1)th frame image based on the frequency domain information of the (i-1)th frame image; The determining module is used to determine that the third virtual object is a newly added virtual object in the (i-1)th frame image when the keyframe image has not been marked with a region corresponding to the third region.

17. The apparatus according to claim 16, characterized in that, The determining module is further configured to, if the difference between the area of ​​the first region of the keyframe image and the area of ​​the second region of the (i-1)th frame image satisfies a threshold, identify the (i-1)th frame image as a new keyframe image.

18. The apparatus according to any one of claims 10 to 14, characterized in that, The marking module is also used to mark the fourth region corresponding to the fourth virtual object in the (i-1)th frame image based on the frequency domain information of the (i-1)th frame image; The acquisition module is also used to acquire the i-th frame image in the video stream based on the virtual environment; The determining module is further configured to determine, in the case that the i-th frame image has not been marked with a region corresponding to the fourth region, that the fourth virtual object is a virtual object reduced from the i-th frame image.

19. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the method for identifying virtual objects as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the method for identifying virtual objects as described in any one of claims 1 to 9.

21. A computer program product, characterized in that, The computer program product includes computer instructions, which are loaded and executed by a processor to implement the method for identifying virtual objects as described in any one of claims 1 to 9.

Images