Player identification method, device and equipment

Abstract

The specification discloses an athlete identification method, device and equipment, which comprises the following steps: obtaining three-dimensional human body grids of each athlete and interfering personnel from video image sequences containing two opposing sides with clothing colors complying with the regulations of the sports scene; extracting vertex color values from the video image according to vertex visibility states and grid projection relationships, and mapping the vertex color values to a fixed-resolution UV map through a pre-defined vertex index to generate a gesture-invariant human body texture representation; and then classifying the UV map through a pre-trained classification model to obtain the identification result of the two opposing sides in the video image. Since the classification model is trained for the clothing features of the two opposing sides, the background (including referees and spectators) interference can be accurately excluded; and the UV map maps the clothing local features to fixed positions, which can realize fine extraction of human body clothing, exclude human body gesture interference, and improve the identification accuracy.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of computer vision, and in particular to a player identification method, device and equipment. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence and computer vision technology, intelligent analysis of sports videos has become an important application direction in the field of sports technology. By automatically identifying and analyzing elements such as characters, actions, and scenes in sports videos, technical teams can provide deeper data support for coaches, athletes, and spectators. These technologies are widely used in event broadcasting, tactical analysis, player performance evaluation, injury prevention, and other fields, improving the fairness and watchability of competitions, and also promoting the intelligentization of athlete training.

[0003] Taking the boxing project as an example, the identification of red and blue boxers is a basic step for subsequent intelligent analysis of techniques and tactics. Unlike football, tennis and other projects, the boxing competition field is small and close to the audience, and the referee is also on the ring with the boxer, making it difficult to exclude these background characters by adjusting the shooting angle. Using a multi-target tracking algorithm to analyze the competition video will track all characters. Therefore, an algorithm is needed to identify the opposing players from the background characters in a two-against-sports scene. SUMMARY

[0004] In view of this, one or more embodiments of the present specification provide a player identification method, device and equipment to improve the accuracy of identifying players in a two-against-sports scene.

[0005] To achieve the above object, the technical solutions of one or more embodiments of the present specification are as follows:

[0006] According to a first aspect of one or more embodiments of the present specification, a player identification method is provided, comprising:

[0007] Obtaining a video image sequence containing a sports scene of two opposing parties, wherein the clothing colors of the two opposing parties conform to the regulations of the sports scene;

[0008] Obtaining a three-dimensional human mesh of each player and interfering personnel from the image frames in the sequence, respectively, wherein the three-dimensional human mesh contains vertex three-dimensional coordinates and vertex visibility state;

[0009] According to the vertex visibility state and the mesh projection relationship, extracting vertex color values from the video images;

[0010] Mapping the color values of each mesh vertex to a fixed resolution UV map through a predefined vertex index to generate a pose-invariant human texture representation;

[0011] inputting the UV map into a pre-trained classification model to obtain a classification result, wherein the classification model is trained based on a UV map data set in the sports scene;

[0012] identifying the players and the interference personnel in the image frame according to the classification result.

[0013] In some embodiments, the three-dimensional human body mesh of each player and interference personnel is obtained from the image frames in the sequence, respectively, including:

[0014] For the image frames, a human body detection box and a segmentation mask of each player and interference personnel are obtained according to a human body detection algorithm or a tracking algorithm;

[0015] A human body region image is cropped according to the human body detection box, and a pre-trained human body pose estimation model is used to process the human body region image to obtain vertex three-dimensional coordinates and camera parameters;

[0016] The vertex three-dimensional coordinates are projected to a two-dimensional image plane by using the camera parameters;

[0017] The visibility state of the vertex is determined according to the projected coordinates and the segmentation mask.

[0018] In some embodiments, the visibility state of the vertex is determined according to the projected coordinates and the segmentation mask, including:

[0019] For each vertex, if the projected coordinates are located in a region with a segmentation mask of 1 and are not blocked by other targets, the vertex is marked as a visible state;

[0020] Otherwise, the vertex is marked as an invisible state.

[0021] In some embodiments, the vertex color value is extracted from the video image according to the visibility state of the vertex and the mesh projection relationship, including:

[0022] For a vertex with a visible state, the color value of the position corresponding to the two-dimensional projected coordinates is extracted from the video image;

[0023] For a vertex with an invisible state, the color value is set to zero, or the vertex color value is determined according to the color value of the vertex in the visible state in other frames within a time window of the current frame.

[0024] In some embodiments, the vertex color value is determined according to the color value of the vertex in the visible state in other frames within a time window of the current frame, including:

[0025] In the case of real-time processing, the vertex color value is determined according to the color value of the vertex in the visible state in the nearest occurrence before the current frame.

[0026] In the case of offline processing, the vertex color value is determined according to the color value of the vertex in a visible state in the last occurrence after the current frame; or

[0027] The vertex color value is determined according to the average value of the color values of the vertex in a visible state in all occurrences within a set time window.

[0028] In some embodiments, the mapping of the coordinates and color values of each mesh vertex to a fixed resolution UV map comprises:

[0029] Mapping the three-dimensional mesh vertex to a UV plane by a predefined mapping function, wherein the UV plane is discretized into a rasterized map;

[0030] For each raster position, at least one vertex mapped to the position is obtained;

[0031] The raster is filled according to the color value corresponding to the at least one vertex.

[0032] In some embodiments, the filling of the raster according to the color value corresponding to the at least one vertex comprises at least one of:

[0033] According to the distance weight of the two-dimensional projection position of each vertex and the center of the raster, a fused color value is generated to fill the raster;

[0034] According to the color value of the vertex closest to the two-dimensional projection position of each vertex and the center of the raster, the raster is filled.

[0035] In some embodiments, the method further comprises:

[0036] Constructing a UV map dataset containing three categories of the opposing parties and the interfering personnel;

[0037] Training the classification model using the UV map dataset.

[0038] According to a second aspect of one or more embodiments of the present specification, a player recognition device is provided, comprising:

[0039] A first acquisition unit is configured to acquire a video image sequence of a sports scene containing opposing parties, wherein the clothing color of the opposing parties conforms to the regulations of the sports scene;

[0040] A second acquisition unit is configured to obtain a three-dimensional human body mesh of each player and interfering personnel from the image frames in the sequence, respectively, wherein the three-dimensional human body mesh contains vertex three-dimensional coordinates and vertex visibility state;

[0041] extracting, by an extraction unit, vertex color values from the video image according to the vertex visibility state and a mesh projection relationship;

[0042] mapping, by a mapping unit, the color values of the vertices of each mesh to a UV map of a fixed resolution via a predefined vertex index, to generate a pose-invariant human texture representation;

[0043] inputting, by a classification unit, the UV map into a pre-trained classification model to obtain a classification result, wherein the classification model is trained based on a UV map dataset in the motion scene;

[0044] identifying, by an identification unit, the opposing players in the image frame according to the classification result.

[0045] In some embodiments, the second acquisition unit is specifically configured to:

[0046] obtaining, for the image frame, a human detection box and a segmentation mask of each player and interfering personnel according to a human detection algorithm or a tracking algorithm;

[0047] cropping a human region image according to the human detection box and processing the human region image via a pre-trained human pose estimation model to obtain vertex three-dimensional coordinates and camera parameters;

[0048] projecting the vertex three-dimensional coordinates to a two-dimensional image plane via the camera parameters;

[0049] determining a vertex visibility state according to the projected coordinates and the segmentation mask.

[0050] In some embodiments, when determining the vertex visibility state according to the projected coordinates and the segmentation mask, the second acquisition unit is specifically configured to:

[0051] for each vertex, marking as a visible state if the projected coordinates are located in a region with a segmentation mask of 1 and are not occluded by other targets;

[0052] otherwise, marking as an invisible state.

[0053] In some embodiments, the extraction unit is specifically configured to:

[0054] for a vertex with a visible state, extracting a color value of a position corresponding to the two-dimensional projected coordinates from the video image;

[0055] for a vertex with an invisible state, setting the color value to zero or determining the vertex color value according to color values of the vertex in visible states in other frames within a time window of the current frame.

[0056] In some embodiments, the extraction unit is specifically configured to determine the vertex color value according to a color value of the vertex in a visible state in other frames within a set time window according to the current frame, and specifically configured to:

[0057] In the case of real-time processing, the vertex color value is determined according to a color value of the vertex in a visible state in the last occurrence before the current frame;

[0058] In the case of offline processing, the vertex color value is determined according to a color value of the vertex in a visible state in the last occurrence after the current frame; or

[0059] The vertex color value is determined according to an average of color values of the vertex in a visible state in all occurrences within a set time window.

[0060] In some embodiments, the mapping unit is specifically configured to: map the three-dimensional mesh vertex to a UV plane through a predefined mapping function, wherein the UV plane is discretized into a rasterized map; for each raster position, obtain at least one vertex mapped to the position; and fill the raster according to color values corresponding to the at least one vertex.

[0061] In some embodiments, the mapping unit is specifically configured to, when filling the raster according to the color values corresponding to the at least one vertex, generate a fused color value filled raster according to a distance weight of a two-dimensional projection position of each vertex from a raster center; and fill the raster according to a color value of a vertex closest to the raster center in a two-dimensional projection position of each vertex.

[0062] In some embodiments, the apparatus further comprises a training unit configured to: construct a UV map dataset containing three categories of adversarial parties and interference personnel; and train the classification model using the UV map dataset.

[0063] According to a third aspect of one or more embodiments of the present specification, an electronic device is provided, comprising:

[0064] a processor;

[0065] a memory for storing processor-executable instructions;

[0066] wherein the processor implements the steps of the method proposed in the above embodiments by running the executable instructions.

[0067] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, which stores computer instructions that, when executed by a processor, implement the steps of the method proposed in the above embodiments.

[0068] According to a fifth aspect of one or more embodiments of the present specification, a computer program product is provided, comprising computer programs / instructions which, when executed by a processor, implement the steps of the method according to the above embodiments.

[0069] The player recognition method according to the embodiments of the present specification obtains a three-dimensional human body mesh of each player and an interfering person from a video image sequence containing two opposing players with clothing colors complying with the regulations of the sports scene; extracts vertex color values from the video image according to the vertex visibility state and the mesh projection relationship, and maps the vertex color values to a fixed resolution UV map through a pre-defined vertex index to generate a pose-invariant human body texture representation; then classifies the UV map through a pre-trained classification model to obtain the recognition result of the two opposing players in the video image. Since the classification model is trained for the clothing features of the two players, the background interference (including referees and spectators) can be accurately excluded; and the UV map maps the local features of the clothing to a fixed position, which can achieve fine extraction of the clothing of the human body and exclude the interference of the human body posture, thereby improving the recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 FIG. 1 is a flowchart of a player recognition method according to an example embodiment.

[0071] Figure 2 FIG. 2 is a flowchart of a method for obtaining a three-dimensional human body mesh according to an example embodiment.

[0072] Figure 3 FIG. 3 is a schematic diagram of a UV map sample according to an example embodiment.

[0073] Figure 4 FIG. 4 is a block diagram of a player recognition device according to an example embodiment.

[0074] Figure 5 FIG. 5 is a structural schematic diagram of an apparatus according to an example embodiment. DETAILED DESCRIPTION

[0075] The example embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following example embodiments are not meant to represent all implementations consistent with one or more embodiments of the present specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of the present specification as detailed in the appended claims.

[0076] It should be noted that the steps of the corresponding method are not necessarily performed in the order shown and described in the specification in other embodiments. In some other embodiments, the steps included in the method thereof can be more or less than described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments; while multiple steps described in the specification can be combined into a single step for description in other embodiments.

[0077] Figure 1 A flowchart of a player identification method provided by an embodiment of the application is shown, which comprises:

[0078] Step 101, acquiring a video image sequence containing a sports scene of two opposing players.

[0079] The sports scene containing two opposing players refers to a competitive event in which two players directly confront each other in a limited space and are distinguished by their clothing colors, such as boxing, wrestling, judo, etc. In these sports scenes, the clothing colors of the two players are in accordance with the specified requirements of the sport, for example, one side wears red and the other side wears blue in a boxing match.

[0080] The video image sequence can be a sequence of consecutive images in a video captured by a camera, or a real-time video stream.

[0081] Step 102, acquiring a three-dimensional human mesh of each player and interfering personnel from the image frames in the sequence, respectively, the three-dimensional human mesh containing vertex three-dimensional coordinates and vertex visibility state.

[0082] For each frame in the video image sequence, a target detection model (such as Mask-RCNN) or a tracking algorithm (DeepSORT algorithm) can be used to obtain the human body detection box and segmentation mask of each player and interfering personnel, so as to determine the three-dimensional human mesh of each player and interfering personnel detected. The three-dimensional human mesh is a digital human surface model used to represent the body shape and position of the human body in three-dimensional space. Its core elements include vertices and topological structure, and in addition to three-dimensional coordinates, vertices also have visibility state information. Visibility state is a binary label that marks whether each vertex can be observed in the current frame image. In the visible state, the body part corresponding to the vertex is not occluded and located within the image range, while in the invisible state, the vertex cannot be observed due to occlusion or beyond the field of view.

[0083] Step 103, extracting vertex color values from the video image according to the vertex visibility state and mesh projection relationship.

[0084] The mesh projection relationship refers to the geometric mapping relationship between the three-dimensional human body mesh vertex and the two-dimensional video image plane, which is converted by the spatial coordinates established by the camera model. For the vertex with a visible state of visible, the color value of the corresponding position in the two-dimensional projection coordinates is extracted from the video image; for the vertex with a visible state of invisible, the color value is set to zero, or the vertex color value is determined according to the color value of the vertex in the visible state in other frames within the time window of the current frame.

[0085] Step 104, the color value of each mesh vertex is mapped to a fixed resolution UV map through a predefined vertex index, generating a pose-invariant human body texture representation.

[0086] The predefined vertex index mapping refers to the establishment of a fixed mapping relationship between the human body topology and the texture plane by binding the ID of each vertex of the three-dimensional mesh to the fixed UV coordinates. In the case of changes in the human body pose, the coordinates of the three-dimensional mesh vertex also change, however, the fixed mapping of the UV map through the vertex index is independent of the human body pose, which can eliminate the texture position changes caused by rotation, translation, scaling and other changes, and the same body area (such as the chest) has consistent position in the UV map of different players.

[0087] In addition, the texture details can still be clearly presented under the fixed resolution. That is, the dynamic human body is converted into a static texture representation, which lays a foundation for subsequent player recognition.

[0088] For example, the vertices of the human body mesh can be mapped to a fixed size UV map by constructing a UV mapping function. Assuming that the size of the UV map is DxD, the UV mapping function is u,v=f(j), u,v∈[1,D], and then the color information of the human body mesh is mapped to the UV map. For the visible vertices on the human body mesh, the corresponding position on the UV map is the color of the vertex in the video frame, and for the invisible vertices, the RGB value of the corresponding position on the UV map is set to 0; if the adjustment value has been set for the invisible vertex, the RGB value of the corresponding position on the UV map is set to the adjustment value, and finally the UV map is obtained.

[0089] Step 105, inputting the UV map into a pre-trained classification model to obtain a classification result.

[0090] The classification model is trained based on the UV map dataset in the sports scene. The dataset provides standardized UV map samples of players and interfering personnel for the classification model. Taking a boxing match as an example, the classification model provides samples of red players, blue players and interfering personnel, as shown in FIG. 2. Figure 3

[0091] ​In one example, the structure of the UV map classification model can be as follows: first, a ResNet-18 model is used to extract visual features, and then a linear layer is used to convert the multi-dimensional output of the ResNet-18 to three dimensions to realize three classification. Those skilled in the art should understand that the above structure is only an example, and other classification model structures can also be used, and the present disclosure does not intend to limit this.

[0092] Step 106, according to the classification result, identifying the opposing players in the image frame.

[0093] In the embodiments of the present disclosure, from the video image sequence containing the opposing players with clothing colors conforming to the regulations of the sports scene, three-dimensional human body meshes of each player and interfering personnel are obtained respectively; according to the vertex visibility state and the mesh projection relationship, the vertex color value is extracted from the video image, and is mapped to the UV map with fixed resolution through the pre-defined vertex index, to generate the posture-invariant human body texture representation; then the pre-trained classification model is used to classify the UV map to obtain the identification result of the opposing players in the video image. Since the classification model is trained for the clothing features of the opposing players, the background interference (including referees and spectators) can be accurately excluded; and the UV map maps the local features of the clothing to the fixed position, which can realize the fine extraction of the clothing of the human body and exclude the interference of the human body posture, to improve the recognition accuracy.

[0094] In some embodiments, the three-dimensional human body mesh of each player and interfering personnel can be obtained by using the method as shown in Figure 2 The method includes:

[0095] Step 201, for the image frame, the human body detection box (bbox) and the segmentation mask (mask) of each player and interfering personnel are obtained according to the human body detection algorithm or the tracking algorithm.

[0096] Specifically, for the input video sequence, the architecture of human body detection initialization, tracking maintenance and periodic correction can be used. That is, for the initial frame, the human body detection algorithm can be used to execute, and then the tracking algorithm is used to maintain, and in the case of not meeting the tracking condition, the human body detection algorithm is re-executed, so as to optimize the calculation efficiency while ensuring the accuracy.

[0097] By executing the human body detection algorithm or the tracking algorithm on one image frame in the video image sequence, the human body detection box detection and the human body segmentation mask extraction can be realized, and the formula is as follows.

[0098]

[0099] Wherein, F represents the human body detection model, V irepresents the i-th frame of the game video, K represents the number of people detected in the frame.

[0100] In step 202, the human region image is cropped according to the human detection frame, and the human region image is processed by the pre-trained human pose estimation model to obtain the vertex three-dimensional coordinates and camera parameters.

[0101] According to the detected bbox, each human region detected is cropped, and a human mesh estimation is performed using a human pose estimation model (for example, DensePose-RCNN), and the formula is as follows.

[0102]

[0103] wherein H represents the human pose estimation model, bbox k represents the bbox of the k-th detected person, crop represents the picture cropping operation according to the bbox, verts 3D represents the three-dimensional coordinates of the estimated human mesh vertex, and cam represents the camera parameters of the person, including the translation parameter t and the scaling parameter s.

[0104] In step 203, the vertex three-dimensional coordinates are projected to the two-dimensional image plane using the camera parameters.

[0105] By the camera parameters and the human mesh vertex three-dimensional coordinates, the two-dimensional projection of the human mesh vertex in the image is obtained, verts 2D = projection(verts 3D , cam).

[0106] In step 204, the vertex visibility state is determined according to the projection coordinates and the segmentation mask.

[0107] For each vertex, if the projection coordinates are located in the region where the segmentation mask is 1 and are not blocked by other targets, it is marked as a visible state; otherwise, it is marked as an invisible state.

[0108] Specifically, according to the two-dimensional coordinates of the human mesh vertex and the mask output of the detection model, the color information of the visible part of the human body in the image frame is mapped to the human mesh. Assuming that the 2D coordinates of the j-th vertex projected on the image are verts 2D,j = [h j , w j ]: if the mask value mask[h j , w j ] at the coordinates is 1, the color value V i [h j , w jThe jth vertex of the human body mesh is set to be invisible if the mask value mask[h j ,w j ] at the coordinate is 0.

[0109] In some embodiments, the vertex color value can be determined according to the color value of the vertex in the visible state in other frames within the time window set according to the current frame in the following way.

[0110] In the case of real-time processing, the vertex color value is determined according to the color value of the vertex in the visible state in the frame that appears most recently before the current frame, that is, the vertex color value is determined using the forward nearest visible frame.

[0111] Specifically, for a vertex v that is invisible in the current frame t, the frame k (k < t) that is nearest to t before t is found, in which the vertex v is visible, and the color value of v in the frame k is used.

[0112] In the case of offline processing, the vertex color value can be determined according to the color value of the vertex in the visible state in the frame that appears most recently after the current frame, that is, the vertex color value is determined according to the backward nearest visible frame.

[0113] Specifically, for a vertex v that is invisible in the current frame t, the frame m (m > t) that is nearest to t after t is found, in which the vertex v is visible, and the color value of v in the frame m is used.

[0114] The vertex color value can also be determined according to the average of the color values of the vertex in the visible state in all frames within the set time window.

[0115] Specifically, for a vertex v that is invisible in the current frame t, within a fixed time window [t-N, t+M] centered at t (or containing only before t), the color values in the frames in which the vertex v is in the visible state are collected, and the average of the RGB channels of these color values is calculated as the color value of v in the frame t.

[0116] In the embodiments of the present disclosure, through multi-frame information fusion, the limitations of single-frame vision can be broken through, a time dimension observation model can be established, and the robustness, accuracy and stability of the estimation of the color value of the invisible vertex can be improved.

[0117] In some embodiments, the mapping of the coordinates and color values of each mesh vertex to a UV map of a fixed resolution comprises: mapping the three-dimensional mesh vertex to a UV plane through a predefined mapping function, wherein the UV plane is discretized into a rasterized map; for each grid position, obtaining at least one vertex mapped to the position; and filling the grid according to the color value corresponding to the at least one vertex.

[0118] In some cases, such as the crease area of a judo uniform, or the curved surface of a boxing glove, there can be multiple vertices mapped to the same grid, and a certain strategy is needed to determine the color value filled in.

[0119] In one example, a distance-first strategy can be adopted, that is, filling the grid with the color value of the vertex closest to the center of the grid according to the two-dimensional projection position of each vertex. This strategy is suitable for scenarios with high requirements for computing efficiency.

[0120] In another example, a distance-weighted strategy can be adopted, that is, generating a fused color value to fill the grid according to the distance weight of the two-dimensional projection position of each vertex from the center of the grid. This strategy is suitable for scenarios with high requirements for texture quality and visual continuity.

[0121] Figure 4 is a block diagram of an athlete identification device provided by an example embodiment. As shown in Figure 4 the device includes:

[0122] A first acquisition unit 401 is configured to acquire a video image sequence containing a sports scene of two opposing parties, wherein the clothing colors of the two opposing parties conform to the regulations of the sports scene.

[0123] A second acquisition unit 402 is configured to acquire three-dimensional human body meshes of each athlete and interfering personnel from image frames in the sequence, respectively, wherein the three-dimensional human body meshes contain vertex three-dimensional coordinates and vertex visibility states.

[0124] An extraction unit 403 is configured to extract vertex color values from the video images according to the vertex visibility states and mesh projection relationships.

[0125] A mapping unit 404 is configured to map the color values of each mesh vertex to a fixed resolution UV map through a predefined vertex index, to generate a pose-invariant human body texture representation.

[0126] A classification unit 405 is configured to input the UV map into a pre-trained classification model to obtain a classification result, wherein the classification model is trained based on a UV map dataset in the sports scene.

[0127] An identification unit 406 is configured to identify the athletes of the two opposing parties in the image frames according to the classification result.

[0128] In some embodiments, the second acquisition unit is specifically configured to:

[0129] For the image frames, human body detection boxes and segmentation masks of each athlete and interfering personnel are obtained according to a human body detection algorithm or a tracking algorithm.

[0130] cropping a human region image from the human bounding box, and processing the human region image by a pre-trained human pose estimation model to obtain vertex three-dimensional coordinates and camera parameters;

[0131] projecting the vertex three-dimensional coordinates to a two-dimensional image plane by using the camera parameters;

[0132] determining a vertex visibility state according to the projected coordinates and the segmentation mask.

[0133] In some embodiments, the second obtaining unit, when used for determining the vertex visibility state according to the projected coordinates and the segmentation mask, is specifically used for:

[0134] for each vertex, marking as a visible state in a case that the projected coordinates are located in a region with a segmentation mask of 1 and are not occluded by other targets;

[0135] otherwise, marking as an invisible state.

[0136] In some embodiments, the extracting unit is specifically used for:

[0137] for a vertex with a visibility state of visible, extracting a color value of a position corresponding to the two-dimensional projected coordinates from the video image;

[0138] for a vertex with a visibility state of invisible, setting the color value to zero, or determining the vertex color value according to color values of the vertex in visible states in other frames within a time window of the current frame.

[0139] In some embodiments, the extracting unit, when used for determining the vertex color value according to the color values of the vertex in visible states in other frames within a time window of the current frame, is specifically used for:

[0140] in a case of real-time processing, determining the vertex color value according to a color value of the vertex in a visible state in a nearest time of the current frame;

[0141] in a case of offline processing, determining the vertex color value according to a color value of the vertex in a visible state in a nearest time after the current frame; or

[0142] determining the vertex color value according to an average value of color values of the vertex in visible states in all appearances within a set time window.

[0143] In some embodiments, the mapping unit is specifically used for:

[0144] mapping the three-dimensional mesh vertex to a UV plane by using a pre-defined mapping function, wherein the UV plane is discretized into a rasterized map;

[0145] for each raster position, obtaining at least one vertex mapped to the position.

[0146] fill the grid according to the color value corresponding to the at least one vertex.

[0147] In some embodiments, the mapping unit, when filling the grid according to the color value corresponding to the at least one vertex, is specifically configured to:

[0148] fill the grid with the fusion color value according to the distance weight of the two-dimensional projection position of each vertex from the center of the grid;

[0149] fill the grid with the color value of the vertex closest to the center of the grid according to the two-dimensional projection position of each vertex.

[0150] In some embodiments, the device further comprises a training unit configured to:

[0151] construct a UV map data set containing three categories of the two opposing parties and the interfering personnel;

[0152] train the classification model using the UV map data set.

[0153] Figure 5 is a schematic structural diagram of a device provided by an exemplary embodiment. Please refer to Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and of course can also include other hardware required by the business. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 510 into the memory 508 by the processor 502 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0154] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0155] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memories.

[0156] Memory can include, without being limited to, non- transitory storage in computer-readable media, random access memory (RAM), and / or read-only memory (ROM), such as flash memory, etc. Memory is an example of computer-readable media.

[0157] Computer-readable media includes permanent and non- permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, without being limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic disks storage, quantum memory, graphene-based storage media, or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition provided herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0158] It is also important to note that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0159] The above description of certain examples of the application has been presented for the purposes of illustration and description. Other examples are within the scope and range of equivalents of the claims. In some cases, acts or steps can be performed in an order different from that of the examples, and / or at least partially concurrently, and still accomplish the desired results. Also, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0160] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments described in this description. As used in this description and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be understood that the terms "and / or," "comprises" and / or "comprising" when used in this description and the appended claims are used to describe one or more of the claimed features. It will be further understood that terms, such as those defined in the Dictionary of

[0161] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, the information should not be limited to these terms. These terms are used only to distinguish one particular information from another. For example, a first information could be termed a second information, and, similarly, a second information could be termed a first information without departing from the scope of one or more embodiments described herein. As used herein, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" taking into account the context in which the term is used.

[0162] The foregoing is merely illustrative of the principles of one or more embodiments described herein and various modifications can be made by persons skilled in the art without departing from the scope of one or more embodiments described herein.

Claims

1. A contestant identification method, characterized in that, include: Acquire a video image sequence containing a motion scene of opposing sides, wherein the clothing colors of the opposing sides conform to the regulations of the motion scene; The three-dimensional human body mesh of each contestant and the interfering person is obtained from the image frames in the sequence. The three-dimensional human body mesh includes the three-dimensional coordinates of the vertices and the visibility status of the vertices. Based on the vertex visibility state and mesh projection relationship, vertex color values ​​are extracted from the video image; The color values ​​of each mesh vertex are mapped to a UV map of fixed resolution using a predefined vertex index to generate a pose-invariant human texture representation. The UV map is input into a pre-trained classification model to obtain a classification result, wherein the classification model is trained based on the UV map dataset in the motion scene; The opposing players in the image frame are identified based on the classification results.

2. The method according to claim 1, characterized in that, The step of obtaining the three-dimensional human body mesh of each contestant and interfering person from the image frames in the sequence includes: For the image frame, the human detection bounding boxes and segmentation masks of each contestant and interfering person are obtained according to the human detection algorithm or tracking algorithm. The human body region image is cropped according to the human body detection box, and the human body region image is processed by a pre-trained human pose estimation model to obtain the vertex 3D coordinates and camera parameters. The three-dimensional coordinates of the vertices are projected onto a two-dimensional image plane using the camera parameters. Vertex visibility status is determined based on projection coordinates and segmentation mask.

3. The method according to claim 2, characterized in that, The step of determining vertex visibility state based on projection coordinates and segmentation mask includes: For each vertex, if the projected coordinates are located in the region with a segmentation mask of 1 and are not occluded by other targets, it is marked as visible. Otherwise, mark it as invisible.

4. The method according to claim 1, characterized in that, Extracting vertex color values ​​from the video image based on the vertex visibility state and mesh projection relationship includes: For vertices whose visibility is visible, extract the color value of the corresponding position in the two-dimensional projection coordinates from the video image; For vertices whose visibility is not visible, set the color value to zero, or determine the vertex color value based on the color value of the vertex in the visible state in other frames within the current frame's time window.

5. The method according to claim 4, characterized in that, The vertex color value is determined based on the color values ​​of the vertex in the visible state in other frames within the current frame's time window, including: In real-time processing, the vertex color value is determined based on the color value of the vertex in the visible state most recently before the current frame. In offline processing, the vertex color value is determined based on the color value of the vertex in its most recent visible state after the current frame; or The vertex color value is determined based on the average color value of all visible vertices that appear within a set time window.

6. The method according to claim 4, characterized in that, The process of mapping the coordinates and color values ​​of each mesh vertex to a UV map of fixed resolution includes: The vertices of the 3D mesh are mapped to the UV plane using a predefined mapping function, wherein the UV plane is discretized into a rasterized texture. For each grid position, obtain at least one vertex mapped to that position; The grid is filled according to the color value corresponding to the at least one vertex.

7. The method according to claim 6, characterized in that, The step of filling the raster according to the color value corresponding to the at least one vertex includes at least one of the following: Based on the distance weights between the two-dimensional projection position of each vertex and the center of the grid, a blended color value is generated to fill the grid. The grid is filled with the color value of the vertex whose two-dimensional projection position is closest to the grid center.

8. The method according to claim 1, characterized in that, The method further includes: Construct a UV map dataset that includes three categories: opposing sides and interfering personnel; The classification model is trained using the UV mapping dataset.

9. A contestant identification device, characterized in that, include: The first acquisition unit is used to acquire a video image sequence containing a motion scene of opposing sides, wherein the clothing colors of the opposing sides conform to the regulations of the motion scene; The second acquisition unit is used to acquire the three-dimensional human body mesh of each contestant and the interfering person from the image frames in the sequence, and the three-dimensional human body mesh includes the three-dimensional coordinates of the vertices and the visibility status of the vertices. An extraction unit is used to extract vertex color values ​​from the video image based on the vertex visibility state and the mesh projection relationship; The mapping unit is used to map the color values ​​of each mesh vertex to a UV map of fixed resolution through a predefined vertex index, thereby generating a pose-invariant human texture representation. A classification unit is used to input the UV map into a pre-trained classification model to obtain a classification result, wherein the classification model is trained based on the UV map dataset in the motion scene; The identification unit is used to identify the two opposing players in the image frame based on the classification result.

10. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-8 by executing the executable instructions.

Images