A multi-modal fusion-based sports tactics information retrieval method and system

Abstract

The application provides a kind of based on multi-modal fusion's movement tactic information retrieval method, including obtaining the movement trajectory of input movement target;With movement tactic database, the final movement tactic is obtained by similarity measurement;Wherein, by measuring the similarity between the movement trajectory of the input movement target and the true value movement trajectory in the movement tactic database, the highest similarity is obtained, that is, the final movement tactic.The movement tactic database includes manually annotated tactical label and multi-modal fusion movement trajectory.The retrieval method of the application has the characteristics of general and more accurate retrieval, the method combines multi-modal movement trajectory information, can correct the significant error of a certain mode, realize the accurate retrieval of each target movement tactic, the method adds tactical registration, and the data of the tactical database can be expanded.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a movement tactic information retrieval method and system based on multi-modal fusion. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] At present, with the advent of deep learning networks, computer vision has played an irreplaceable role in various fields and industries. In the field of artificial intelligence + sports, deep learning algorithms can be seen everywhere. In the Olympic Games, the diving team uses 3D pose estimation to analyze the diving and entry posture of each athlete, which helps the coach to better correct and standardize the movements of each team member. Similarly, in the sports competition, training and teaching of basketball and other scenes, the coach needs to analyze the movement trajectory of each player on the court, summarize and explain. Currently, some methods use video information and related computer technology for tactic retrieval. These methods only input video, and the input data mode is single. In addition, the information provided by the video information shot from a fixed angle is limited. At present, sensor, unmanned aerial vehicle and other hardware facilities are developed, and it is not difficult to obtain the drawn movement trajectory of the player on the court. The coach will also hand-draw the trajectory analysis tactic of the player on the court. However, the current movement tactic retrieval and recognition method only uses video data as the input data of movement tactic retrieval, and does not consider multi-modal information. Moreover, the current movement retrieval and recognition method does not explain how to obtain the movement trajectory of the player through the video, does not fuse the trajectory information drawn by the video mode and other modes, and cannot comprehensively use external information for player tactic retrieval, so as to help the coach to make better decisions. SUMMARY

[0004] In order to solve the above problems, the present application provides a movement tactic information retrieval method and system based on multi-modal fusion. The present application can fuse the trajectory information drawn by the video mode and other modes, and comprehensively use external information for player tactic retrieval.

[0005] According to some embodiments, the present application adopts the following technical scheme:

[0006] A movement tactic information retrieval method based on multi-modal fusion, comprising:

[0007] obtaining the movement trajectory of the input movement target;

[0008] using the movement tactic database to obtain the final movement tactic through similarity measurement;

[0009] Wherein, by measuring the similarity between the motion trajectory of the input motion target and the true value motion trajectory in the motion tactics database, the highest similarity is obtained, that is, the final motion tactics.

[0010] Further, the motion tactics database includes manually annotated tactics labels and multi-modal fusion motion trajectories.

[0011] Further, the multi-modal fusion motion trajectory includes, first, obtaining a motion video of a fixed perspective and preprocessing to obtain an image sequence.

[0012] Further, the multi-modal fusion motion trajectory also includes, obtaining the prediction box of the motion target in the image sequence by a multi-target tracking algorithm.

[0013] Further, the multi-modal fusion motion trajectory also includes, obtaining the motion trajectory of the motion target in the motion video of the fixed perspective by a trajectory drawing method.

[0014] Further, the multi-modal fusion motion trajectory also includes, mapping the motion trajectory of the motion target in the motion video of the fixed perspective to the basketball court plan by perspective transformation, to obtain the motion trajectory in the basketball court plan.

[0015] Further, the multi-modal fusion motion trajectory also includes, obtaining the multi-modal fusion motion trajectory by averaging the manually annotated tactics labels and the motion trajectory in the basketball court plan.

[0016] A multi-modal fusion-based motion tactics information retrieval system, comprising:

[0017] The data acquisition module is configured to acquire the motion trajectory of the input motion target;

[0018] The motion tactics module is configured to use the motion tactics database to obtain the final motion tactics by similarity measurement.

[0019] Wherein, by measuring the similarity between the motion trajectory of the input motion target and the true value motion trajectory in the motion tactics database, the highest similarity is obtained, that is, the final motion tactics.

[0020] A computer readable storage medium, wherein a plurality of instructions are stored, the instructions are suitable for being loaded and executed by a processor of a terminal device.

[0021] A terminal device, comprising a processor and a computer readable storage medium, the processor is used to implement each instruction; the computer readable storage medium is used to store a plurality of instructions, the instructions are suitable for being loaded and executed by the processor.

[0022] Compared with the prior art, the present application has the beneficial effects of:

[0023] The retrieval method has the characteristics of universality and more accurate retrieval, combines multi-modal motion trajectory information, can correct significant errors of a certain mode, realizes accurate retrieval of each target motion tactic, adds tactic registration to expand the data of the tactic database, in addition, uses perspective transformation to solve the problem of non-uniform view angle of multi-modal information, so that the multi-modal information can be more accurately fused, and the effectiveness of the method is verified through experiments on existing motion retrieval data sets. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings accompanying the specification of this application form a part thereof, serve to further provide a further understanding of the application, and together with the specification explain the application, and do not constitute an improper limitation of the application.

[0025] Figure 1 is an architecture diagram of the embodiment;

[0026] Figure 2 is a motion target trajectory drawing schematic diagram of the embodiment;

[0027] Figure 3 is a flowchart of the embodiment. DETAILED DESCRIPTION

[0028] The application will be further described below in conjunction with the drawings and embodiments.

[0029] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0030] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments, and are not intended to limit the exemplary embodiments according to the application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0031] Embodiment 1.

[0032] As shown in Figure 1 , a motion tactic information retrieval method based on multi-modal fusion includes:

[0033] Obtaining the motion trajectory of the input motion target;

[0034] The final sports tactics are obtained by similarity measurement using the sports tactics database.

[0035] The similarity between the movement trajectory of the input sports target and the true value movement trajectory in the sports tactics database is measured, and the one with the highest similarity is the final sports tactic.

[0036] The sports tactics database includes manually labeled tactical labels and multi-modal fusion movement trajectories.

[0037] The multi-modal fusion movement trajectory includes first obtaining a fixed perspective sports video and pre-processing to obtain an image sequence.

[0038] The multi-modal fusion movement trajectory also includes obtaining the prediction box of the movement target in the image sequence through a multi-target tracking algorithm.

[0039] The multi-modal fusion movement trajectory also includes obtaining the movement trajectory of the movement target in the fixed perspective sports video through a trajectory drawing method.

[0040] The multi-modal fusion movement trajectory also includes mapping the movement trajectory of the movement target in the fixed perspective sports video to the basketball court plan through perspective transformation to obtain the movement trajectory in the basketball court plan.

[0041] The multi-modal fusion movement trajectory also includes obtaining the multi-modal fusion movement trajectory by averaging the manually labeled tactical labels and the movement trajectory in the basketball court plan.

[0042] Specifically,

[0043] The method comprises:

[0044] Step S0: video data V preprocessing, obtaining N frames of images, denoted as Where f i represents the i-th frame of the obtained video;

[0045] Step S1: using a multi-target tracking algorithm to process the image sequence obtained in step S0 M movement targets and the tracking detection box coordinates of each target can be obtained;

[0046] Step S2: according to the coordinate information obtained in step S1, the movement trajectories of the M movement targets can be drawn. For example, Figure 2As shown, assuming that the detection box coordinate information of the jth target in the ith frame is (x1, y1, w1, h1), and the detection box coordinate information in the i+1th frame is (x2, y2, w2, h2), the trajectory of the target in the continuous frames is a vector with the starting point (x1+w1 / 2, y1+h1) and the ending point (x2+w2 / 2, y2+h2), that is, the black arrow vector in the figure, and the trajectories in the continuous frames are connected, so that the motion trajectory of each target in the video can be obtained;

[0047] Step S3: mapping the M motion trajectories obtained in step S2 to the basketball court plan, and the specific process is as follows: the transformation matrix required for mapping can be obtained by using the perspective transformation function in the opencv library, and by using the transformation matrix and the image transformation function in the opencv library, the target motion trajectory obtained in step S2 in the shooting angle can be mapped to the basketball court plan;

[0048] Step S4: manually drawing each target motion trajectory and labeling the tactical label on the basketball court plan, and the label data is denoted as

[0049] Step S5: establishing a coordinate system on the basketball court plan, so that the coordinates of each target trajectory obtained in step S3 and the coordinates of the manually drawn target trajectory obtained in step S4 can be obtained. Taking the average of the two coordinates, the new corrected motion target trajectory is obtained, and the coordinates of each target motion trajectory are denoted as , and the label is still

[0050] Step S6: the trajectory coordinate data obtained in step S5 is registered with the label data obtained in step S4 , that is, the tactical information is stored in the database label file;

[0051] Step S7: when performing tactical retrieval, assuming that the manually drawn trajectory or the video drawn trajectory coordinate of an input is , the input trajectory is compared with the motion tactical trajectory in the tactical database to measure the similarity, and the highest similarity is the motion tactical of the target, so that the motion tactical retrieval of the target is realized, and the index for measuring the similarity is the Euclidean distance, denoted as d, and the coordinates on the true value of the motion tactical trajectory in the tactical database are denoted as , and the formula is as follows;

[0052]

[0053] Example 2.

[0054] Figure 3 The algorithm flowchart of the multi-modal fusion motion tactical information retrieval method of the application specifically includes the following steps:

[0055] Step S0: Select any video data preprocessing in dataset A to get 1000 frames of images, denoted as Where f i represents the i-th frame of the obtained video;

[0056] Step S1: Process the image sequence obtained in step S0 using the deepsort algorithm Get which contains 5 moving targets, and the tracking detection box coordinates of each target are obtained;

[0057] Step S2: Draw the motion trajectory of the 5 moving targets in the video according to the coordinate information. Assuming that the detection box coordinate information of the first target in the 50th frame is (50, 120, 64, 128), and the detection box coordinate information in the i+1th frame is (54, 121, 64, 128), the trajectory of the target in the continuous frames is a vector with the starting point (82, 248) and the ending point (86, 249). Connecting the trajectories in several continuous frames, the motion trajectory of each target in the video can be obtained;

[0058] Step S3: Map the 5 motion trajectories obtained in step S2 to the basketball court plan, the specific process is as follows: the transformation matrix required for mapping can be obtained using the perspective transformation function in the opencv library. Using the transformation matrix, combined with the image transformation function in the opencv library, the target motion trajectory obtained in step S2 can be mapped to the basketball court plan from the shooting angle;

[0059] Step S4: Manually draw the motion trajectory of each target on the basketball court plan and label its tactical label, and the label data is denoted as {dribbling, pick and roll, passing, running, shooting};

[0060] Step S5: Establish a coordinate system on the basketball court plan, so as to obtain the coordinates of each target trajectory obtained in step S3 and the coordinates of the manually drawn target trajectory obtained in step S4. Take the average of the two coordinates to get the new corrected motion target trajectory. Assuming that the coordinates of each target motion trajectory are The label is still {dribbling, pick and roll, passing, running, shooting};

[0061] Step S6: The trajectory coordinate data obtained in step S5 is registered with the label data {dribbling, pick and roll, passing, running, shooting} obtained in step S4, and the tactical information is stored in the database label file;

[0062] Step S7: When searching for tactics, the manually drawn trajectory or video drawn trajectory coordinate is input The input trajectory is compared with the movement tactical trajectory in the tactical database, the highest similarity is the movement tactical of the target, so as to realize the movement tactical retrieval of the target, the similarity is measured by the Euclidean distance, denoted as d, and the coordinates on the true value of the movement tactical trajectory in the tactical database are denoted as , and the formula is as follows.

[0063]

[0064] In this embodiment, our method is not used, only video data is used for movement target trajectory drawing, and then movement tactical identification is performed, and the identification accuracy is 45%; by using our method, the perspective transformation in steps S3-S5 and the trajectory mean fusion are used, the movement trajectory after multi-modal fusion is obtained for movement tactical identification, the similarity is measured by the Euclidean distance in step S6 for tactical retrieval, and the identification accuracy is 70%; in addition, by using the tactical registration method, the expansion of the data of the tactical retrieval database is realized. By the method, multi-modal trajectory information fusion is introduced for movement tactical retrieval, the multi-modal information is unified to the same movement plane by transformation, and then fusion is performed, so that multi-modal is realized, and then movement tactical retrieval is performed. More dimensional trajectory information is provided, so that the method achieves high accuracy in the existing movement tactical retrieval task.

[0065] Embodiment 3.

[0066] A movement tactical information retrieval system based on multi-modal fusion, comprising:

[0067] A data acquisition module configured to acquire the movement trajectory of an input movement target;

[0068] A movement tactical module configured to use a movement tactical database to obtain the final movement tactical by similarity measurement.

[0069] The similarity between the movement trajectory of the input movement target and the true value movement trajectory in the movement tactical database is measured, and the highest similarity is the final movement tactical.

[0070] Embodiment 4.

[0071] A computer readable storage medium, wherein a plurality of instructions are stored, the instructions are suitable for being loaded and executed by a processor of a terminal device to perform a power emergency material scheduling and configuration method provided in the embodiment.

[0072] Embodiment 5.

[0073] A terminal device comprises a processor and a computer readable storage medium, the processor is used to implement instructions; the computer readable storage medium is used to store a plurality of instructions, the instructions are suitable for being loaded by the processor and executing a power emergency material scheduling and configuration method provided by the embodiment.

[0074] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0075] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus produce a device implemented in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0076] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction apparatus, which implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0077] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0078] The above merely provides preferred embodiments of the present application, but is not intended to limit the present application. Based on the above teachings, one skilled in the art will be able to implement various modifications and variations without departing from the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of the present application.

[0079] The above merely provides preferred embodiments of the present application, but is not intended to limit the present application. Based on the above teachings, one skilled in the art will be able to implement various modifications and variations without departing from the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of the present application.

Claims

1. A method for searching sports tactics information based on multi-modal fusion, characterized in that, include: Obtain the motion trajectory of the input moving target; utilize the motion tactics database and obtain the final motion tactics through similarity measurement; Specifically, the motion tactic is determined by measuring the similarity between the motion trajectory of the input moving target and the ground truth motion trajectory in the motion tactic database. The motion tactic database includes manually labeled tactical tags and multimodal fused motion trajectories. The multimodal fused motion trajectory is obtained by mapping the motion trajectory of the moving target in the motion video from a fixed perspective using perspective transformation onto a basketball court plan view. After obtaining the motion trajectory in the basketball court plan view, the manually labeled tactical tags and the motion trajectory in the basketball court plan view are then fused using the mean.

2. The multi-modal fusion-based sports tactical information retrieval method of claim 1, wherein, The motion trajectory of the multimodal fusion includes first acquiring motion video from a fixed perspective and preprocessing it to obtain an image sequence.

3. The method for retrieving motion tactical information based on multimodal fusion as described in claim 2, characterized in that, The motion trajectory of the multimodal fusion also includes obtaining the predicted bounding box of the moving target in the image sequence through a multi-target tracking algorithm.

4. The method for retrieving motion tactical information based on multimodal fusion as described in claim 3, characterized in that, The multimodal fusion motion trajectory also includes obtaining the motion trajectory of a moving target in a motion video from a fixed perspective through a trajectory drawing method.

5. A motion tactical information retrieval system based on multimodal fusion, characterized in that, include: The data acquisition module is configured to acquire the motion trajectory of the input moving target; The sports tactics module is configured to use a sports tactics database to obtain the final sports tactics through similarity measurement; Specifically, the motion tactic is determined by measuring the similarity between the motion trajectory of the input moving target and the ground truth motion trajectory in the motion tactic database. The motion tactic database includes manually labeled tactical tags and multimodal fused motion trajectories. The multimodal fused motion trajectory is obtained by mapping the motion trajectory of the moving target in the motion video from a fixed perspective using perspective transformation onto a basketball court plan view. After obtaining the motion trajectory in the basketball court plan view, the manually labeled tactical tags and the motion trajectory in the basketball court plan view are then fused using the mean.

6. A computer-readable storage medium, characterized in that, It stores multiple instructions, which are adapted to be loaded and executed by the processor of the terminal device as any one of claims 1-4, a method for retrieving motion tactical information based on multimodal fusion.

7. A terminal device, characterized in that, The device includes a processor and a computer-readable storage medium, wherein the processor implements various instructions; and the computer-readable storage medium stores multiple instructions adapted to be loaded and executed by the processor, as described in any one of claims 1-4, for retrieving motion tactical information based on multimodal fusion.

Images