Virtual sports system based on computer vision and implementation method thereof

Abstract

The embodiment of the invention discloses a virtual sports system based on computer vision and an implementation method thereof, and relates to the technical field of machine vision. The method comprises the following steps: acquiring first motion data generated when a user responds to an initial interactive interface of a target virtual sports scene; generating a first interaction interface based on the first motion data; responding to the first motion data, and obtaining interaction action response data; generating a second interaction interface according to the interaction action response data; acquiring second motion data generated when the user responds to the second interaction interface, continuously feeding back and responding to the second motion data, and ending the virtual sports interaction until a preset condition is met; wherein the first motion data and the second motion data are obtained by splicing outputs obtained by processing motion videos collected for a user through the human body key point detection sub-model and the spatial-temporal feature extraction sub-model.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular, to a virtual sports system based on computer vision and an implementation method thereof. Background Art

[0002] A virtual sports system is a system that uses computer vision technology to capture the user's motion posture through a camera, and combines artificial intelligence algorithms to analyze, process and respond to motion data, thereby enabling users to exercise and compete in a virtual environment. The system can simulate sports scenes, provide a highly interactive and interesting virtual sports experience, and provide real-time feedback and evaluation of the user's sports performance.

[0003] Currently, the user's movements can be identified by identifying human joints, and then the identified movements are fed back to the virtual sports system for related control. However, since the user is active during use, some joints may be blocked due to factors such as the image acquisition perspective, which affects the accuracy of movement recognition and related control results. In addition, when processing motion data, existing virtual sports systems often lack in-depth analysis of user movement details, which may ultimately lead to a lack of realism in the virtual sports experience.

[0004] Based on this, the present application proposes a virtual sports system based on computer vision and its implementation method, aiming to improve the accuracy of virtual sports scene control and user experience through more accurate human motion data recognition and analysis. Summary of the invention

[0005] In order to solve the above technical problems, one aspect of the embodiments of this specification provides a method for implementing a virtual sports system based on computer vision, the method comprising:

[0006] Acquiring first motion data generated by a user's response to an initial interactive interface of a target virtual sports scene;

[0007] Feeding back the first motion data to the target virtual sports scene, and generating a first interactive interface based on the first motion data;

[0008] In response to the first motion data, acquiring interactive action response data;

[0009] generating a second interaction interface according to the interaction action response data;

[0010] Acquire second motion data generated by the user's response to the second interactive interface, and continuously provide feedback and respond to the second motion data until the virtual sports interaction is terminated when a preset condition is met;

[0011] Among them, the first motion data and the second motion data are obtained by processing the motion video collected for the user through a human motion data recognition model, and the human motion data recognition model includes a human key point detection submodel and a spatiotemporal feature extraction submodel. The first motion data and the second motion data are obtained by splicing the outputs of the human key point detection submodel and the spatiotemporal feature extraction submodel.

[0012] In some embodiments, before obtaining the first motion data generated by the user's response to the initial interactive interface of the target virtual sports scene, the method also includes: constructing the target virtual sports scene according to the type of sports activity selected by the user and the environmental characteristics of the user's environment.

[0013] In some embodiments, the step of constructing a target virtual sports scene according to the type of sports activity selected by the user and the environmental characteristics of the environment where the user is located includes:

[0014] Acquire a plurality of preset scene pictures corresponding to the sports activity type selected by the user;

[0015] Capturing images of the environment where the user is located, and extracting corresponding environmental features;

[0016] Calculating the similarity between each of the preset scene images and the environment where the user is located according to the environmental characteristics;

[0017] A preset scene picture with the highest similarity to the environment where the user is located is selected as the target scene picture, and the target virtual sports scene is constructed based on the target scene picture.

[0018] In some embodiments, the human motion data recognition model includes a preprocessing unit, and the preprocessing unit is used to preprocess the motion video to obtain a preprocessed video frame, wherein the preprocessing includes at least denoising and / or enhancement;

[0019] The step of obtaining first motion data generated by a user responding to an initial interactive interface of a target virtual sports scene includes:

[0020] Inputting the preprocessed video frames obtained after preprocessing each video frame in the motion video into the human key point detection submodel and the spatiotemporal feature extraction submodel respectively, obtaining human key point data output by the human key point detection submodel and spatiotemporal feature data output by the spatiotemporal feature extraction submodel, wherein the human key point data includes the position coordinates of the human key points in the corresponding video frame, and the spatiotemporal feature data includes a high-dimensional spatiotemporal feature vector for reflecting the human motion pattern in the motion video;

[0021] The human body key point data and the spatiotemporal feature data are spliced ​​to obtain the first motion data.

[0022] In some embodiments, the first interaction interface includes a first motion object corresponding to or related to the user, and generating the first interaction interface based on the first motion data includes:

[0023] The first motion data is mapped to the target virtual sports scene through a preset first motion data mapping rule, so as to control a first motion object corresponding to or related to the user in the first interactive interface.

[0024] In some embodiments, mapping the first motion data to the target virtual sports scene by using a preset first motion data mapping rule to control a first motion object corresponding to or related to the user in the first interactive interface includes:

[0025] The first target preset motion data closest to the first motion data is searched in the mapping table corresponding to the first motion data mapping rule, and the first motion object corresponding to or related to the user in the first interactive interface is controlled according to the first target preset motion data.

[0026] In some embodiments, in response to the first motion data, acquiring interactive action response data includes:

[0027] Respond to the first motion data based on a preset interactive action response data mapping rule to obtain at least one candidate response data; then, determine one data from the at least one candidate response data as the interactive action response data;

[0028] Alternatively, motion data generated by the online user in response to the first motion data is obtained as the interactive action response data, wherein the motion data generated by the online user in response to the first motion data is obtained by processing the motion video collected for the online user through the human motion data recognition model.

[0029] In some embodiments, the second interaction interface includes a second motion object related to the user, and generating the second interaction interface according to the interaction action response data includes:

[0030] The interactive action response data is mapped to the target virtual sports scene through a preset second motion data mapping rule to control a second motion object related to the user in the second interactive interface.

[0031] In some embodiments, the continuously providing feedback and responding to the second motion data includes:

[0032] Feeding back the second motion data to the target virtual sports scene, and generating a corresponding first interactive interface based on the second motion data;

[0033] In response to the second motion data, determining corresponding interactive action response data;

[0034] Generate a corresponding second interaction interface according to the interaction action response data corresponding to the second motion data.

[0035] Another aspect of the embodiments of this specification also provides a virtual sports system based on computer vision, the system comprising:

[0036] A motion data acquisition module, used to acquire first motion data generated by a user's response to an initial interactive interface of a target virtual sports scene;

[0037] An interactive interface generation module, used for feeding back the first motion data to the target virtual sports scene, and generating a first interactive interface based on the first motion data;

[0038] An interactive action data acquisition module, used for acquiring interactive action response data in response to the first motion data;

[0039] The interaction interface generation module is also used to generate a second interaction interface according to the interaction action response data;

[0040] The motion data acquisition module is further used to acquire the second motion data generated by the user's response to the second interactive interface, and the interactive interface generation module is further used to continuously feedback and respond to the second motion data until the virtual sports interaction is terminated when a preset condition is met;

[0041] Among them, the first motion data and the second motion data are obtained by processing the motion video collected for the user through a human motion data recognition model, and the human motion data recognition model includes a human key point detection submodel and a spatiotemporal feature extraction submodel. The first motion data and the second motion data are obtained by splicing the outputs of the human key point detection submodel and the spatiotemporal feature extraction submodel.

[0042] The beneficial effects that may be brought about by the virtual sports system based on computer vision and its implementation method provided in the embodiments of this specification include at least:

[0043] (1) By simultaneously using the human key point detection sub-model and the spatiotemporal feature extraction sub-model to process the motion video generated by the user's response to the initial interactive interface of the target virtual sports scene, human key point data reflecting the position coordinates of the human key points in the corresponding video frame, and spatiotemporal feature data containing high-dimensional spatiotemporal feature vectors reflecting the human motion pattern in the motion video can be obtained simultaneously. Furthermore, by splicing the human key point data and the spatiotemporal feature data to obtain the first motion data generated by the user's response to the initial interactive interface of the target virtual sports scene, the spatiotemporal feature data can be used to supplement the motion information when some human key points are blocked and the user's motion cannot be accurately represented, thereby more accurately representing the first motion data generated by the user, and then achieving more precise control in the subsequent process, thereby enhancing the realism of the virtual sports experience;

[0044] (2) By matching similar preset scene images according to the environmental characteristics of the user's environment and constructing the target virtual sports scene based on the matching results, the user's style preferences for the virtual sports scene can be met to a certain extent, thereby improving the user's usage experience;

[0045] (3) By searching for the first target preset motion data that is closest to the first motion data in the mapping table corresponding to the first motion data mapping rule, and controlling the first motion object corresponding to or related to the user in the first interactive interface according to the first target preset motion data, the problem of unresponsive action and discontinuous motion process caused by the inconsistency between the first motion data and all the data in the mapping table corresponding to the first motion data mapping rule can be avoided.

[0046] Additional features will be described in part in the following description. For those skilled in the art, it will become apparent by reviewing the following and accompanying drawings, or it may be understood by the production or operation of the examples. The features of this specification may be implemented and obtained by practicing or using various aspects of the methods, tools, and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0048] Figure 1 is a schematic diagram of an exemplary application scenario of a virtual sports system based on computer vision according to some embodiments of this specification;

[0049] Figure 2is an exemplary module diagram of a virtual sports system based on computer vision according to some embodiments of this specification;

[0050] Figure 3 is an exemplary flow chart of a method for implementing a virtual sports system based on computer vision according to some embodiments of this specification;

[0051] Figure 4 is an exemplary structural diagram of a human motion data recognition model according to some embodiments of this specification;

[0052] Figure 5 It is an exemplary step flow chart of a method for implementing a virtual sports system based on computer vision as shown in other embodiments of this specification. DETAILED DESCRIPTION

[0053] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0054] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0055] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0056] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0057] The computer vision-based virtual sports system and its implementation method provided in the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0058] Figure 1 It is a schematic diagram of an exemplary application scenario of a computer vision-based virtual sports system according to some embodiments of this specification.

[0059] Reference Figure 1 In some embodiments, the application scenario 100 of the computer vision-based virtual sports system may include an image acquisition device 110, a storage device 120, a processing device 130, a display terminal 140, and a network 150. The various components in the application scenario 100 may be connected in a variety of ways. For example, the image acquisition device 110 may be connected to the storage device 120 and / or the processing device 130 through the network 150, or may be directly connected to the storage device 120 and / or the processing device 130. For another example, the storage device 120 may be directly connected to the processing device 130 or connected through the network 150. For another example, the display terminal 140 may be connected to the storage device 120 and / or the processing device 130 through the network 150, or may be directly connected to the storage device 120 and / or the processing device 130.

[0060] In an embodiment of the present application, the image acquisition device 110 can be used to collect motion videos for the user and to collect environmental images for the environment where the user is located. In an embodiment of the present application, the motion video collected by the image acquisition device 110 for the user can reflect the motion characteristics of the user in the process of performing virtual sports. In some embodiments, the image acquisition device 110 may include at least one camera (such as a 3D camera or an ordinary camera). In some embodiments of the present application, in order to achieve multi-angle motion video acquisition, the image acquisition device 110 can be set on the pan-tilt assembly. In some embodiments, the user position can be tracked in real time by a human body tracking algorithm, and the acquisition angle of the image acquisition device 110 can be controlled according to the tracked user position. In some embodiments, the image acquisition device 110 can have an independent power supply, which can send the collected motion video and environmental image to other components (for example, storage device 120, processing device 130, terminal device 140) in the application scene 100 by wired or wireless (for example, Bluetooth, WiFi, etc.). In some embodiments, the application scene 100 may include multiple (for example, two or more) image acquisition devices 110, and the multiple image acquisition devices 110 can collect motion videos for the same or different users.

[0061] In some embodiments, the image acquisition device 110 can send the motion video and / or environmental image it has collected to the storage device 120, the processing device 130, the display terminal 140, etc. through the network 150. In some embodiments, the motion video and / or environmental image collected by the image acquisition device 110 can be processed by the processing device 130. For example, the processing device 130 can determine the motion data of the user based on the motion video. In some embodiments, the motion data of the user can be sent to the storage device 120 for recording, or sent to the display terminal 140 for feedback to the user.

[0062] The network 150 may facilitate the exchange of information and / or data. The network 150 may include any suitable network capable of facilitating the exchange of information and / or data of the application scenario 100. In some embodiments, at least one component of the application scenario 100 (e.g., the image acquisition device 110, the storage device 120, the processing device 130, the display terminal 140) may exchange information and / or data with at least one other component in the application scenario 100 via the network 150. For example, the processing device 130 may obtain a motion video captured for a user from the image acquisition device 110 and / or the storage device 120 via the network 150.

[0063] In some embodiments, the network 150 may be any form of wired or wireless network, or any combination thereof. By way of example only, the network 150 may include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, etc., or any combination thereof. In some embodiments, the network 150 may include at least one network access point, and at least one component of the application scenario 100 may be connected to the network 150 through the access point to exchange data and / or information.

[0064] The storage device 120 may store data, instructions, and / or any other information. In some embodiments, the storage device 120 may store data obtained from the image acquisition device 110, the processing device 130, and / or the display terminal 140. For example, the storage device 120 may store motion video acquired by the image acquisition device 110; for another example, the storage device 120 may store motion data obtained by the processing device 130 according to the aforementioned motion video processing. In some embodiments, the storage device 120 may store data and / or instructions used by the processing device 130 to execute or use to complete the exemplary methods described in this specification. In some embodiments, the storage device 120 may include a large-capacity memory, a removable memory, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof. Exemplary large-capacity memories may include magnetic disks, optical disks, solid-state disks, etc. In some embodiments, the storage device 120 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc., or any combination thereof.

[0065] In some embodiments, the storage device 120 may be connected to the network 150 to communicate with at least one other component in the application scenario 100 (e.g., the image acquisition device 110, the processing device 130, the display terminal 140). At least one component in the application scenario 100 may access data, instructions, or other information stored in the storage device 120 through the network 150. In some embodiments, the storage device 120 may be directly connected or communicate with one or more components in the application scenario 100 (e.g., the image acquisition device 110, the display terminal 140). In some embodiments, the storage device 120 may be part of the image acquisition device 110 and / or the processing device 130.

[0066] The processing device 130 can process data and / or information obtained from the image acquisition device 110, the storage device 120, the display terminal 140 and / or other components of the application scenario 100. In some embodiments, the processing device 130 can obtain a motion video from any one or more of the image acquisition device 110 and the storage device 120, and determine the user's motion data by processing the motion video. In some embodiments, the processing device 130 can obtain a pre-stored computer instruction from the storage device 120, and execute the computer instruction to implement the implementation method of the virtual sports system based on computer vision described in this specification.

[0067] In some embodiments, the processing device 130 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 130 may be local or remote. For example, the processing device 130 may access information and / or data from the image acquisition device 110, the storage device 120, and / or the display terminal 140 via the network 150. For another example, the processing device 130 may be directly connected to the image acquisition device 110, the storage device 120, and / or the display terminal 140 to access information and / or data. In some embodiments, the processing device 130 may be implemented on a cloud platform. For example, a cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud cloud, a multi-cloud, etc., or any combination thereof.

[0068] The display terminal 140 can receive, send and / or display data. The received data may include data collected by the image acquisition device 110, data stored in the storage device 120, data processed by the processing device 130, etc. The sent data may include user operation instructions (such as start, pause, end, etc.). In an embodiment of the present application, the display terminal 140 may include a TV 141, a projector 142, a wearable VR device 143 (such as VR glasses, VR helmets, etc.), etc. In some embodiments, the display terminal 140 may include an input device (such as a button, a touch screen, a microphone), an output device (such as a display, a speaker), etc. In some embodiments, the processing device 130 may be a part of the display terminal 140.

[0069] It should be noted that the above description of the application scenario 100 is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the application scenario 100 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification. For example, the image acquisition device 110 may include more or fewer functional components.

[0070] Figure 2 is a schematic diagram of a module of a virtual sports system based on computer vision according to some embodiments of this specification. In some embodiments, Figure 2 The computer vision-based virtual sports system 200 shown can be applied to Figure 1 The application scenario 100 shown, for example, can be configured in the form of software and / or hardware to the processing device 130 and / or the display terminal 140 to process the sports video captured by the image acquisition device 110, determine the user's sports data based on the sports video, and then perform virtual sports interaction based on the sports data.

[0071] Reference Figure 2In some embodiments, the virtual sports system 200 based on computer vision may include a motion data acquisition module 210, an interactive interface generation module 220, and an interactive action data acquisition module 230. Among them:

[0072] The motion data acquisition module 210 can be used to acquire first motion data generated by a user's response to an initial interactive interface of a target virtual sports scene.

[0073] The interaction interface generation module 220 may be configured to feed back the first motion data to the target virtual sports scene, and generate a first interaction interface based on the first motion data.

[0074] The interactive action data acquisition module 230 may be configured to respond to the first motion data and acquire interactive action response data.

[0075] In an embodiment of the present application, the interactive interface generation module 220 can also be used to generate a second interactive interface based on the interactive action response data. The motion data acquisition module 210 can also be used to obtain the second motion data generated by the user's response to the second interactive interface. The motion data acquisition module 210, the interactive interface generation module 220 and the interactive action data acquisition module 230 can also be used to continuously feedback and respond to the second motion data until the virtual sports interaction is terminated when the preset conditions are met. Among them, the first motion data and the second motion data are obtained by processing the motion video collected for the user through a human motion data recognition model, and the human motion data recognition model includes a human key point detection submodel and a spatiotemporal feature extraction submodel. The first motion data and the second motion data are obtained by splicing the outputs of the human key point detection submodel and the spatiotemporal feature extraction submodel.

[0076] For more details about the above modules, please refer to other places in this manual (for example Figure 3 to Figure 5 part and its related description), which will not be repeated here.

[0077] It should be understood that Figure 2The computer vision-based virtual sports system 200 and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or included in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of this specification can not only be implemented by hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software such as executed by various types of processors, and can also be implemented by a combination of the above hardware circuits and software (e.g., firmware).

[0078] It should be noted that the above description of the virtual sports system 200 based on computer vision is provided for illustrative purposes only and is not intended to limit the scope of this specification. It is understandable that for those skilled in the art, according to the description of this specification, without departing from this principle, the various modules can be arbitrarily combined, or a subsystem can be formed to connect with other modules. For example, Figure 2 The motion data acquisition module 210, the interactive interface generation module 220, and the interactive action data acquisition module 230 described in the embodiment can be different modules in a system, or a module can realize the functions of two or more of the above modules. Such variations are within the protection scope of this specification. In some embodiments, the aforementioned modules can be part of the processing device 130 and / or the display terminal 140.

[0079] Figure 3 is an exemplary flow chart of a method for implementing a virtual sports system based on computer vision according to some embodiments of this specification. In some embodiments, the method for implementing a virtual sports system based on computer vision can be performed by processing logic, which can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (instructions running on a processing device to perform hardware simulation), etc., or any combination thereof. In some embodiments, Figure 3 One or more operations in the flowchart of the method for implementing the computer vision-based virtual sports system can be performed by Figure 1For example, the method for implementing a virtual sports system based on computer vision may be stored in the storage device 120 in the form of a computer program and / or instruction, and may be called and / or executed by the processing device 130 .

[0080] Reference Figure 3 In an embodiment of the present application, a method for implementing a virtual sports system based on computer vision may include:

[0081] Step S310 , obtaining first motion data generated by a user's response to an initial interactive interface of a target virtual sports scene. In some embodiments, step S310 may be performed by the motion data obtaining module 210 .

[0082] In some embodiments, a sports video of a user responding to an initial interactive interface of a target virtual sports scene may be acquired through the image acquisition device 110. The target virtual sports scene refers to sports activities performed by the user in a virtual environment, such as basketball, volleyball, tennis, table tennis, running, cycling, etc.

[0083] The initial interaction interface of the target virtual sports scene can be understood as the interface that the user sees before starting the virtual sports activity, which provides a starting point for the user to interact. For example, for virtual tennis, the initial interaction interface may indicate that the current user or the opponent (which may be an online player or a virtual character controlled by a computer) serves. If the interface indicates that the current user serves, the initial interaction interface may refer to the instruction interface. If the interface indicates that the opponent serves, the instruction interface and the display interface corresponding to the opponent's serving process can be regarded as the initial interaction interface.

[0084] In the embodiment of the present application, the actions performed by the user on the initial interactive interface can be collected by the image acquisition device 110, thereby obtaining a motion video of the user responding to the initial interactive interface of the target virtual sports scene. Further, the motion video can be input into the human motion data recognition model for processing, thereby obtaining the first motion data generated by the user responding to the initial interactive interface of the target virtual sports scene.

[0085] Figure 4 is an exemplary structural diagram of a human motion data recognition model according to some embodiments of this specification. Figure 4As shown, in an embodiment of the present application, the human motion data recognition model may include a preprocessing unit, a human key point detection submodel, a spatiotemporal feature extraction submodel and a splicing unit. Among them, the preprocessing unit can be used to preprocess the motion video to obtain a preprocessed video frame. In an embodiment of the present application, the preprocessing process may at least include denoising and / or enhancement. The human key point detection submodel can be used to process the preprocessed video frame to obtain human key point data, and the human key point data can include the position coordinates of the human key points in the corresponding video frame. The spatiotemporal feature extraction submodel can be used to process the preprocessed video frame to obtain spatiotemporal feature data, and the spatiotemporal feature data can include a high-dimensional spatiotemporal feature vector for reflecting the human motion mode in the motion video. The splicing unit can be used to splice the human key point data output by the human key point detection submodel and the spatiotemporal feature data output by the spatiotemporal feature extraction submodel to obtain the motion data corresponding to the motion video.

[0086] Specifically, in some embodiments of the present application, the preprocessing unit can use filtering techniques (such as Gaussian filtering, median filtering, etc.) to remove image noise (such as noise caused by factors such as low light and camera jitter) in the motion video, while retaining key features in each video frame image. In some embodiments, for the temporal noise introduced by the difference between frames in the motion video, timing filtering, motion smoothing and other techniques can also be used for denoising, thereby reducing interference between dynamic objects. In some embodiments, the preprocessing unit can perform contrast enhancement or brightness adjustment on the motion video to enhance the details of each video frame image, especially in low light or high contrast scenes, thereby helping feature extraction and human key point recognition in subsequent processes.

[0087] In some embodiments of the present application, the human key point detection submodel may include existing human key point detection models such as OpenPose, HRNet, PoseResNet, etc., which can detect various key points of the human body in the motion video (such as head, shoulders, arms, elbows, palms, torso, waist, hips, knees, ankles, feet, etc.), and obtain the human key point data corresponding to the motion video according to the position coordinates of each key point in each video frame contained in the motion video in the corresponding video frame. In other words, in an embodiment of the present application, the human key point data output by the human key point detection submodel can be a multi-dimensional time series data, which contains the position coordinate information of each human key point in the corresponding video frame reflected by the video frame corresponding to each moment.

[0088] It should be noted that the above-mentioned OpenPose, HRNet, PoseResNet and other human key point detection models can be regarded as existing technologies. Therefore, their training process and specific application process will not be discussed in detail in this specification.

[0089] In some embodiments of the present application, the spatiotemporal feature extraction submodel may include a 3D convolutional neural network (3DCNN or C3D), which can extract spatiotemporal feature data (i.e., feature data of the time dimension + feature data of the space dimension) from a video frame sequence to capture the dynamic changes of human body movements. By processing the above-mentioned motion video through the 3D convolutional neural network, the spatiotemporal feature data corresponding to the motion video can be obtained, and the spatiotemporal feature data may include a high-dimensional spatiotemporal feature vector for reflecting the human body motion pattern in the motion video.

[0090] Specifically, in the process of processing the above-mentioned motion video through a 3D convolutional neural network, a 3*3*3 three-dimensional convolution kernel can be applied to a video data block of size h*w*t (h is the height of the video frame image, w is the width of the video frame image, and t is the time axis). In the convolution operation, the 27 pixel values ​​in each 3*3*3 cubic area covered by the convolution kernel are multiplied by the 27 weights of the convolution kernel and summed to generate a single pixel value in the corresponding new feature map. Subsequently, the convolution kernel can slide on the height, width and time axis of the video data according to the set step size, and repeat the above operation to obtain the spatiotemporal feature data corresponding to the motion video. It should be pointed out that by using a 3D convolutional neural network to process the motion video, the changing characteristics of the video content over time can be effectively captured, thereby further enriching the spatiotemporal representation capability of the model, and being able to more effectively capture the motion and dynamic features in the motion video.

[0091] In an embodiment of the present application, the 3D convolutional neural network may include multiple convolutional layers, pooling layers, and fully connected layers (for example, in some embodiments, 8 convolutional layers, 5 pooling layers, and 2 fully connected layers may be included), and finally, the 3D convolutional neural network may include a softmax output layer. In some embodiments, the convolution kernel size may be fixed to 3*3*3, with a stride of 1, and different convolutional layers may include the same or different numbers of convolution kernels. More details about the 3D convolutional neural network (such as the training process) may be regarded as prior art and will not be discussed in detail in this specification.

[0092] Reference Figure 4In the embodiment of the present application, by inputting the preprocessed video frames into the human key point detection submodel and the spatiotemporal feature extraction submodel for processing, the human key point data output by the human key point detection submodel and the spatiotemporal feature data output by the spatiotemporal feature extraction submodel can be obtained respectively. Further, in the embodiment of the present application, the human key point data output by the human key point detection submodel and the spatiotemporal feature data output by the spatiotemporal feature extraction submodel can be spliced ​​by a splicing unit to obtain the above-mentioned first motion data.

[0093] It can be understood that in the embodiment of the present application, by simultaneously using the human key point detection submodel and the spatiotemporal feature extraction submodel to process the motion video generated by the user's response to the initial interactive interface of the target virtual sports scene, human key point data for reflecting the position coordinates of the human key points in the corresponding video frame, and spatiotemporal feature data containing high-dimensional spatiotemporal feature vectors for reflecting the human motion pattern in the motion video can be obtained at the same time. In the embodiment of the present application, the first motion data generated by the user's response to the initial interactive interface of the target virtual sports scene is obtained by splicing the human key point data and the spatiotemporal feature data. When some human key points are blocked and the user's action cannot be accurately represented, the action information can be supplemented by the spatiotemporal feature data, thereby more accurately representing the first motion data generated by the user, and then more precise control can be achieved in the subsequent process, thereby enhancing the realism of the virtual sports experience.

[0094] In some embodiments, the first motion data obtained through the above process may be stored in the storage device 120 , and the motion data acquisition module 210 may acquire the first motion data from the storage device 120 .

[0095] In some embodiments, the target virtual sports scene can be a sports scene in which multiple people (two or more people) participate together, for example, double tennis (each side has two members). In this case, the user can include multiple characters, and the image acquisition device 110 can simultaneously perform motion video acquisition of the multiple characters (which can be acquired by one or more cameras), and obtain the first motion data corresponding to each person by processing the motion video. In other words, in some embodiments of the present application, when the target virtual sports scene can be a sports scene in which multiple people (two or more people) participate together, the above-mentioned first motion data can include motion data obtained by processing the motion videos of multiple members.

[0096] Figure 5 is an exemplary step flow chart of a method for implementing a virtual sports system based on computer vision according to other embodiments of this specification. Figure 5In some embodiments of the present application, the target virtual sports scene may be constructed by the following steps S301 to S304 before step S310:

[0097] Step S301, obtaining a plurality of preset scene pictures corresponding to the sports activity type selected by the user.

[0098] In the embodiment of the present application, the user can select the type of sports activity before starting the virtual sports activity, such as basketball, football, volleyball, tennis, table tennis, running, cycling, etc. The system can retrieve the scene images that match the selected sports activity type from the preset scene library according to the user's selection. These scene images can be real stadium environments or virtually designed scenes, aiming to provide users with an immersive virtual sports experience.

[0099] In the embodiment of the present application, different types of sports activities can have preset scene pictures of the same or different styles. For example, basketball and football activities may share a modern stadium scene, while volleyball and tennis may use a seaside sports center scene with different styles. In addition, the system can also provide customized scene design services according to the specific needs of users. For example, users can select specific background music, weather effects or audience interaction elements to enhance the experience of virtual sports activities.

[0100] Step S302: collect images of the environment where the user is located, and extract corresponding environmental features.

[0101] In the embodiment of the present application, the image acquisition device 110 may be used to acquire images of the environment in which the user is located, and then the acquired environment image may be processed by the processing device 130 to extract corresponding environment features.

[0102] Specifically, in the embodiment of the present application, the environmental features of the environmental image may include color, texture, background content, lighting conditions, etc. In some embodiments, the processing device 130 may determine the main environmental features according to the type of sports activity selected by the user. For example, if the user selects a basketball activity, the processing device 130 may focus on identifying and extracting features related to the basketball court, such as the floor texture, color, and background content of the user's environment; for football activities, the system may focus on features such as color and lighting conditions of the user's environment; for tennis activities, it may focus on background content features of the user's environment, etc.

[0103] In an embodiment of the present application, the above environmental features can be automatically identified and classified by a machine learning algorithm. For example, in some embodiments, a convolutional neural network (CNN) model in deep learning can be used to effectively extract complex environmental features from an image. In some embodiments, the processing device 130 can analyze the environmental image using a trained CNN model to identify environmental features associated with different sports activities. In addition, in some embodiments of the present application, in order to improve the accuracy of feature extraction, the system can also be combined with image segmentation technology to distinguish different areas in the environmental image so as to more accurately extract specific environmental features associated with sports activities.

[0104] Step S303: Calculate the similarity between each of the preset scene images and the environment where the user is located according to the environmental characteristics.

[0105] After determining the environmental features of the user's environment through the above steps, the similarity between each preset scene picture and the user's environment can be calculated based on the environmental features. Specifically, in some embodiments of the present application, the environmental features of the user's environment can be represented as a multidimensional feature vector, and a multidimensional feature vector can also be obtained by performing feature extraction on each of the preset scene pictures. In an embodiment of the present application, the similarity between each of the preset scene pictures and the user's environment can be obtained by calculating the distance (e.g., Euclidean distance) between the multidimensional feature vectors.

[0106] Step S304: selecting a preset scene picture with the highest similarity to the environment where the user is located as a target scene picture, and constructing a target virtual sports scene based on the target scene picture.

[0107] In an embodiment of the present application, a preset scene picture with the highest similarity to the user's environment can be selected as a target scene picture, and then a target virtual sports scene can be constructed based on the target scene picture.

[0108] Specifically, in some embodiments, each preset scene picture may correspond to a preset virtual sports scene. After the target scene picture is determined, the construction of the target virtual scene may be completed by reading the virtual sports scene corresponding to the target scene picture.

[0109] In the embodiment of the present application, the target virtual scene may include sports equipment and props required for the type of sports activity selected by the user. For example, if the user selects a basketball activity, the target virtual scene will include a basketball stand, a basketball, and corresponding field markings. For a football activity, it may include a football field, a goal, and a football.

[0110] In the embodiment of the present application, the target virtual scene can be set with corresponding sports rules and parameters, including but not limited to game time, scoring system, foul determination, etc. The system will set corresponding sports rules and parameters according to the standard rules of the selected sports activity, but can also allow users to make personalized adjustments to suit players of different levels or special game modes.

[0111] After completing the construction of the target virtual scene, the user role and motion data can be initialized. In sports activities involving multiple people, the system will create a virtual role for each user, and these virtual roles can be controlled by the user performing corresponding actions in the real world. The specific implementation process of the user controlling the virtual role can be referred to below, and will not be repeated here.

[0112] It can be understood that the environmental characteristics of the user's environment can reflect the user's preference for the scene decoration style to a certain extent. In the embodiment of the present application, by matching similar preset scene images according to the environmental characteristics of the user's environment and constructing the target virtual sports scene based on the matching results, the user's style preference for the virtual sports scene can be satisfied to a certain extent, thereby improving the user's usage experience.

[0113] Continue to refer to Figure 3 After the above step S310, the method for implementing the virtual sports system based on computer vision further includes:

[0114] Step S320 , feeding back the first motion data to the target virtual sports scene, and generating a first interactive interface based on the first motion data. In some embodiments, step S320 may be performed by the interactive interface generation module 220 .

[0115] After obtaining the first motion data generated by the user's response to the initial interactive interface of the target virtual sports scene through the above steps, the first motion data can be fed back to the target virtual sports scene, and a first interactive interface can be generated based on the first motion data.

[0116] In an embodiment of the present application, the first interactive interface includes a first moving object corresponding to or related to the user. It should be noted that, in an embodiment of the present application, the first interactive interface may be a first perspective or a third perspective. The object related to the user in the first interactive interface may refer to an object affected by the user's motion control, such as a basketball in a virtual basketball sport, a tennis ball in a virtual tennis sport, etc. The object corresponding to the user in the first interactive interface may refer to an object directly controlled by the user, such as the virtual character body corresponding to the user, or part of the limbs of the virtual character corresponding to the user (mainly for the first perspective).

[0117] In some embodiments of the present application, the first motion data can be mapped to the target virtual sports scene through a preset first motion data mapping rule, so as to control the first motion object corresponding to or related to the user in the first interactive interface. Among them, the first motion data mapping rule can be understood as a mapping table, which defines the correspondence between different motion data and the first motion object in the virtual sports scene. For example, in a virtual basketball game, when the user performs a jump shooting action, the corresponding motion data will be captured and converted into a basketball shooting action. In an embodiment of the present application, the setting of the first motion data mapping rule can ensure that the user's real motion can be accurately reflected in the virtual sports scene, thereby providing a more realistic and immersive experience. The specific configuration of the first motion data mapping rule can be set accordingly according to actual needs, and this specification will no longer explain and specifically limit it in detail.

[0118] Considering that in some embodiments there may be a situation where the first motion data is inconsistent with all the data in the mapping table corresponding to the first motion data mapping rule, in an embodiment of the present application, in order to ensure that the system can also respond accordingly to the actions performed by the user in this situation, thereby ensuring the continuity of the virtual motion process, the first target preset motion data closest to the first motion data can be searched from the mapping table corresponding to the first motion data mapping rule, and the first motion object corresponding to or related to the user in the first interactive interface can be controlled according to the first target preset motion data.

[0119] Specifically, in some embodiments of the present application, the distance between the first motion data and each motion data in the mapping table corresponding to the first motion data mapping rule can be determined by a Euclidean distance calculation formula, and then the motion data with the closest distance is selected as the first target preset motion data, and the first motion object corresponding to or related to the user in the first interactive interface is controlled according to the first target preset motion data.

[0120] In some embodiments of the present application, the first motion data mapping rule can be dynamically adjusted. This means that the system can automatically adjust the mapping rule according to the user's sports habits or skill level, so that the motion of the motion object in the virtual sports scene is more consistent with the user's actual performance. For example, for beginners, the system may reduce the difficulty of a successful shot, while for advanced users, the difficulty may be increased to maintain the challenge and fun of the virtual sport (also known as a game).

[0121] In addition, the first interactive interface may also include sports objects corresponding to or related to other users or virtual characters, so that users can interact with other users or virtual characters in virtual sports scenes. This interaction can be competitive, such as competition or confrontation, or cooperative, such as collaboration in team sports. In this way, the virtual sports system provided by the present application can not only provide personal sports experience, but also simulate social interactions in the real world, further enhancing the user's sense of participation and satisfaction.

[0122] It should be noted that, in the embodiment of the present application, the first motion data may be continuous, and the motion object corresponding to the user in the first interactive interface may be controlled in real time by the continuous first motion data. As for the motion object related to the user in the first interactive interface, it is necessary to determine whether the first motion data will have an impact on it, and only when the first motion data will have an impact on it, it will be controlled accordingly according to the first motion data. In the embodiment of the present application, it is possible to determine whether the first motion data will have an impact on the motion object related to the user by configuring a preset rule.

[0123] It should be noted that, in the embodiment of the present application, the first interactive interface should be understood as a series of interactive interfaces generated by the system in response to the first motion data corresponding to the user, rather than an interactive interface corresponding to a specific moment.

[0124] Step S330 , in response to the first motion data, acquiring interactive action response data. In some embodiments, step S330 may be executed by the interactive action data acquisition module 230 .

[0125] In some embodiments of the present application, a user can perform virtual sports activities with a computer-controlled virtual character. In this case, the interactive action response data can be understood as control data for controlling the computer virtual character generated by responding to the first motion data of the user after obtaining the first motion data of the user. In this case, the first motion data can be responded to based on a preset interactive action response data mapping rule to obtain at least one candidate response data; then, one data is determined from the at least one candidate response data as the interactive action response data.

[0126] The interactive action response data mapping rule can be understood as a mapping table, which contains the correspondence between multiple sports data and interactive action response data. For example, when the user performs a slice action through the first operation data during the virtual tennis game, the system can generate interactive action response data for the slice action according to the interactive action response data mapping rule to determine the interactive action response data for controlling the computer virtual character in the next step, and then control the computer virtual character based on the interactive action response data.

[0127] In some embodiments of the present application, the same first motion data can be mapped to at least one candidate response data through the interactive action response data mapping rule. In other words, at least one candidate action can be used to respond to the same control action of the user, thereby improving the diversity of the virtual sports process to a certain extent. In some embodiments, a data can be randomly determined from the at least one candidate response data as the interactive action response data, or a data can be determined from the at least one candidate response data as the interactive action response data by round selection.

[0128] In an embodiment of the present application, the interactive action response data mapping rule can be pre-set or dynamically generated by a machine learning algorithm. It should be noted that the mapping rules dynamically generated by the machine learning algorithm can be self-adjusted and optimized according to the user's sports habits and skill level. For example, if the system detects that the user often uses the slice action in the virtual tennis game, and the success rate of the action is high, the system will automatically adjust the mapping rules so that under similar sports data, the computer virtual character is more inclined to perform the slice action. This dynamic adjustment mechanism can make virtual sports activities more personalized, can better simulate the real sports experience, and improve the user's sense of participation and satisfaction.

[0129] In some embodiments of the present application, the user may also perform virtual sports activities with other online users. At this time, the motion data generated by the online user in response to the first motion data may be obtained as the interactive action response data. In an embodiment of the present application, the motion data generated by the online user in response to the first motion data is obtained by processing the motion video collected for the online user through the human motion data recognition model. The specific acquisition method can refer to the aforementioned first motion data acquisition process, which will not be repeated here.

[0130] In some embodiments, the target virtual sports scene may also be a single-person sport, such as running, cycling, etc. In this case, the interactive action response data may refer to control data related to scene changes (such as position changes, movement direction changes, etc.) that may occur in the target virtual sports scene after the user performs the action corresponding to the first motion data. In an embodiment of the present application, the control data can be mapped through a pre-configured mapping table.

[0131] Step S340 , generating a second interactive interface according to the interactive action response data. In some embodiments, step S340 may be performed by the interactive interface generation module 220 .

[0132] After obtaining the interactive action response data through the above steps, the interactive interface generation module 220 can generate a second interactive interface according to the interactive action response data. In this specification, the second interactive interface and the aforementioned first interactive interface can be the same interface or different interfaces. The second interactive interface can include a second motion object related to the user, and the interactive interface generation module 220 can map the interactive action response data to the target virtual sports scene through a preset second motion data mapping rule, thereby controlling the second motion object related to the user in the second interactive interface.

[0133] According to the above content, in the embodiment of the present application, the second moving object may include a virtual character controlled by a computer, a virtual character corresponding to other online users, and a scene that may change with the actions performed by the user.

[0134] Specifically, when the user performs virtual sports activities with a computer-controlled virtual character, the interactive interface generation module 220 can control the computer-controlled virtual character in the second interactive interface based on the interactive action response data. When the user performs virtual sports activities with other online users, the interactive interface generation module 220 can control the virtual characters corresponding to the other online users in the second interactive interface based on the interactive action response data. When the user performs single-player sports, the interactive interface generation module 220 can control the scenes in the target virtual sports scene that may change with the actions performed by the user based on the interactive action response data.

[0135] It should be noted that, in the embodiment of the present application, the second motion data mapping rule is the same as the first motion data mapping rule, that is, the generation method of the second interaction interface is similar to the aforementioned first interaction interface. Therefore, more details about the generation of the second interaction interface can refer to the above content about the first interaction interface, and will not be repeated here.

[0136] Step S350, obtaining the second motion data generated by the user's response to the second interactive interface, and continuously providing feedback and responding to the second motion data until the virtual sports interaction is terminated when a preset condition is met. In some embodiments, step S350 can be performed by the motion data acquisition module 210, the interactive interface generation module 220, and the interactive action data acquisition module 230.

[0137] In the embodiment of the present application, the second motion data generated by the user's response to the second interactive interface can be continuously acquired by the motion data acquisition module 210, and then the corresponding first interactive interface can be generated based on the second motion data by the interactive interface generation module 220. Further, the interactive action data acquisition module 230 can continuously respond to the second motion data and determine the corresponding interactive action response data; finally, the interactive interface generation module 220 can generate the corresponding second interactive interface according to the interactive action response data corresponding to the second motion data.

[0138] In the embodiment of the present application, the above process can be repeated to complete the interaction of virtual sports, until the sports activity is determined to be over according to the corresponding sports rules, and the virtual sports interaction process is terminated.

[0139] For more details about the above-mentioned repetitive process, please refer to the above text and will not be described in detail in this specification.

[0140] In summary, the beneficial effects that may be brought about by the embodiments of the present specification include but are not limited to: (1) In the virtual sports system based on computer vision and its implementation method provided in some embodiments of the present specification, by simultaneously using the human key point detection sub-model and the spatiotemporal feature extraction sub-model to process the motion video generated by the user's response to the initial interactive interface of the target virtual sports scene, human key point data used to reflect the position coordinates of the human key points in the corresponding video frame, and spatiotemporal feature data containing high-dimensional spatiotemporal feature vectors used to reflect the human motion pattern in the motion video can be obtained at the same time. Further, by splicing the human key point data and the spatiotemporal feature data to obtain the first motion data generated by the user's response to the initial interactive interface of the target virtual sports scene, the spatiotemporal feature data can be used to supplement the motion information when some human key points are blocked and the user's motion cannot be accurately represented, thereby more accurately representing the first motion data generated by the user, and then more precise control can be achieved in the subsequent process, thereby improving the realism of the virtual sports experience; (2) In the virtual sports system based on computer vision and its implementation method provided in some embodiments of the present specification, by matching similar preset scene pictures according to the environmental characteristics of the user's environment, and constructing the target virtual sports scene based on the matching results. (3) In the computer vision-based virtual sports system and its implementation method provided in some embodiments of the present specification, by searching the first target preset motion data closest to the first motion data in the mapping table corresponding to the first motion data mapping rule, and controlling the first motion object corresponding to or related to the user in the first interactive interface according to the first target preset motion data, the problem of unresponsive action and discontinuous motion process caused by the inconsistency between the first motion data and all the data in the mapping table corresponding to the first motion data mapping rule can be avoided.

[0141] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or may be any other possible beneficial effects.

[0142] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0143] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0144] In addition, it will be understood by those skilled in the art that various aspects of this specification may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of this specification may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of this specification may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0145] A computer storage medium may include a propagated data signal containing computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination. A computer storage medium may be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit the program for use. The program code on the computer storage medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0146] The computer program code required for the operation of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C language, Visual Basic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages, etc. The program code can be run entirely on the user's computer, or run on the user's computer as an independent software package, or run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or processing device. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0147] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing processing device or mobile device.

[0148] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0149] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0150] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0151] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A method for implementing a virtual sports system based on computer vision, characterized in that: The method comprises: Acquiring first motion data generated by a user's response to an initial interactive interface of a target virtual sports scene; Feeding back the first motion data to the target virtual sports scene, and generating a first interactive interface based on the first motion data; In response to the first motion data, acquiring interactive action response data; generating a second interaction interface according to the interaction action response data; Acquire second motion data generated by the user's response to the second interactive interface, and continuously provide feedback and respond to the second motion data until the virtual sports interaction is terminated when a preset condition is met; Among them, the first motion data and the second motion data are obtained by processing the motion video collected for the user through a human motion data recognition model, and the human motion data recognition model includes a human key point detection submodel and a spatiotemporal feature extraction submodel. The first motion data and the second motion data are obtained by splicing the outputs of the human key point detection submodel and the spatiotemporal feature extraction submodel.

2. The method according to claim 1, characterized in that Before obtaining the first motion data generated by the user's response to the initial interactive interface of the target virtual sports scene, the method also includes: constructing the target virtual sports scene according to the type of sports activity selected by the user and the environmental characteristics of the environment where the user is located.

3. The method according to claim 2, characterized in that The step of constructing a target virtual sports scene according to the type of sports activity selected by the user and the environmental characteristics of the environment where the user is located includes: Acquire a plurality of preset scene pictures corresponding to the sports activity type selected by the user; Capturing images of the environment where the user is located, and extracting corresponding environmental features; Calculating the similarity between each of the preset scene images and the environment where the user is located according to the environmental characteristics; A preset scene picture with the highest similarity to the environment where the user is located is selected as the target scene picture, and the target virtual sports scene is constructed based on the target scene picture.

4. The method according to claim 1, characterized in that The human motion data recognition model includes a preprocessing unit, which is used to preprocess the motion video to obtain a preprocessed video frame, wherein the preprocessing includes at least denoising and / or enhancement; The step of obtaining first motion data generated by a user responding to an initial interactive interface of a target virtual sports scene includes: Inputting the preprocessed video frames obtained after preprocessing each video frame in the motion video into the human key point detection submodel and the spatiotemporal feature extraction submodel respectively, obtaining human key point data output by the human key point detection submodel and spatiotemporal feature data output by the spatiotemporal feature extraction submodel, wherein the human key point data includes the position coordinates of the human key points in the corresponding video frame, and the spatiotemporal feature data includes a high-dimensional spatiotemporal feature vector for reflecting the human motion pattern in the motion video; The human body key point data and the spatiotemporal feature data are spliced ​​to obtain the first motion data.

5. The method according to claim 1, characterized in that The first interaction interface includes a first motion object corresponding to or related to the user, and the generating the first interaction interface based on the first motion data includes: The first motion data is mapped to the target virtual sports scene through a preset first motion data mapping rule, so as to control a first motion object corresponding to or related to the user in the first interactive interface.

6. The method according to claim 5, characterized in that The first motion data is mapped to the target virtual sports scene by a preset first motion data mapping rule to control a first motion object corresponding to or related to the user in the first interactive interface, including: The first target preset motion data closest to the first motion data is searched in the mapping table corresponding to the first motion data mapping rule, and the first motion object corresponding to or related to the user in the first interactive interface is controlled according to the first target preset motion data.

7. The method according to claim 1, characterized in that The acquiring interactive action response data in response to the first motion data includes: Respond to the first motion data based on a preset interactive action response data mapping rule to obtain at least one candidate response data; then, determine one data from the at least one candidate response data as the interactive action response data; Alternatively, motion data generated by the online user in response to the first motion data is obtained as the interactive action response data, wherein the motion data generated by the online user in response to the first motion data is obtained by processing the motion video collected for the online user through the human motion data recognition model.

8. The method according to claim 7, characterized in that The second interaction interface includes a second motion object related to the user, and generating the second interaction interface according to the interaction action response data includes: The interactive action response data is mapped to the target virtual sports scene through a preset second motion data mapping rule to control a second motion object related to the user in the second interactive interface.

9. The method according to claim 1, characterized in that The continuously feeding back and responding to the second motion data includes: Feeding back the second motion data to the target virtual sports scene, and generating a corresponding first interactive interface based on the second motion data; In response to the second motion data, determining corresponding interactive action response data; Generate a corresponding second interaction interface according to the interaction action response data corresponding to the second motion data.

10. A virtual sports system based on computer vision, characterized in that: The system comprises: A motion data acquisition module, used to acquire first motion data generated by a user's response to an initial interactive interface of a target virtual sports scene; An interactive interface generation module, used for feeding back the first motion data to the target virtual sports scene, and generating a first interactive interface based on the first motion data; An interactive action data acquisition module, used for acquiring interactive action response data in response to the first motion data; The interaction interface generation module is also used to generate a second interaction interface according to the interaction action response data; The motion data acquisition module is further used to acquire the second motion data generated by the user's response to the second interactive interface, and the interactive interface generation module is further used to continuously feedback and respond to the second motion data until the virtual sports interaction is terminated when a preset condition is met; Among them, the first motion data and the second motion data are obtained by processing the motion video collected for the user through a human motion data recognition model, and the human motion data recognition model includes a human key point detection submodel and a spatiotemporal feature extraction submodel. The first motion data and the second motion data are obtained by splicing the outputs of the human key point detection submodel and the spatiotemporal feature extraction submodel.

Images