Shuttlecock real-time sensing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and discloses a shuttlecock real-time sensing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a court video stream acquired by image acquisition equipment, the court video stream comprising shuttlecocks in a court; converting the court video stream into a court image; acquiring court images acquired by different image acquisition devices at the same moment, and reconstructing the court images at the same moment to obtain three-dimensional coordinates of the main ball; and determining event information occurring in the court corresponding to the main ball by using a preset event sensing algorithm. The video streams in the court can be comprehensively acquired through the image acquisition device, the badminton can be sensed in real time at low cost, events on the court can be sensed and analyzed without high computing power, manual participation is reduced, different courts can be sensed and analyzed, and the badminton sensing and analyzing efficiency is improved. Therefore, court event sensing analysis can be applied to daily playing scenes, and the use experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a badminton real-time perception method, device, computer equipment and storage medium. Background Art

[0002] The badminton real-time perception and interaction system can use cameras, radars, and other methods to perceive and analyze events on the badminton court, and transmit specific content to different functional modules in real time. Users can use a variety of different terminals to obtain the required information, such as real-time playback, Hawkeye challenge, ball speed measurement, and other functions. Similar technologies are common in professional badminton competitions.

[0003] Among the related technologies, the existing technologies on the market have several technical problems. Existing technical solutions often require many high-speed cameras to capture information on the court, and require high computing power to support the analysis of the content on the court, making the assumptions and operation of the system very costly. At the same time, the existing systems often require a lot of manual participation. These technical problems make it difficult for existing badminton real-time perception and interaction to be applied in daily playing scenarios. Summary of the invention

[0004] In view of this, the present invention provides a badminton real-time perception method, device, computer equipment and storage medium to solve the problem that existing badminton real-time perception and interaction are difficult to apply to daily playing scenes.

[0005] In a first aspect, the present invention provides a badminton real-time sensing method, the method comprising:

[0006] Acquire a court video stream acquired by an image acquisition device, where the court video stream includes badminton in the court;

[0007] Convert the stadium video stream into stadium images, and store the stadium images with the same frame number in the stadium into a queue in the shared memory;

[0008] Obtain court images acquired at the same time by different image acquisition devices from a queue in a shared memory, reconstruct the court images at the same time, and obtain the three-dimensional coordinates of a main ball, where the main ball is used to represent a badminton located in the court;

[0009] Based on the three-dimensional coordinates of the main ball, a preset event perception algorithm is used to determine the event information occurring in the court corresponding to the main ball. The event information is used to characterize the event information occurring in the badminton court at different times.

[0010] In the present invention, the video stream in the court can be fully collected by an image acquisition device, the video stream is converted into an image and the image is reconstructed to obtain the three-dimensional coordinates of the badminton, and the three-dimensional coordinates of the badminton are used to perceive the events occurring in the court. The video stream acquisition can be realized by using an ordinary image acquisition device set in the court, and the real-time perception of the badminton can be realized at a relatively low cost. The perception and analysis of events on the court can be realized without high computing power. At the same time, manual participation can be greatly reduced, and the perception and analysis of events on the court can be realized automatically. It can also realize the perception and analysis of different courts, so that the perception and analysis of court events can be applied to daily playing scenes, which has the potential for large-scale promotion and greatly improves the user experience.

[0011] In an optional implementation, converting the stadium video stream into a stadium image includes:

[0012] Performing time synchronization on the image acquisition device to obtain a time-synchronized image acquisition device;

[0013] The stadium video stream is acquired by using the time-synchronized image acquisition device, and the stadium video stream is converted into a stadium image through GPU hard decoding.

[0014] In this method, in order to obtain accurate badminton reconstruction results, the image acquisition devices are time synchronized so that there is no time difference between the image acquisition devices, or the time difference is within an allowable range, which facilitates the subsequent three-dimensional reconstruction of the court image obtained by converting the video stream captured by the time-synchronized image acquisition device.

[0015] In an optional implementation, reconstructing the stadium image at the same time includes:

[0016] Use the object detection algorithm to detect the court image at the same time and obtain the position coordinates of all badminton balls in each frame of the court image;

[0017] Perform three-dimensional reconstruction on the badminton position coordinates to obtain reconstruction results corresponding to the badminton position coordinates;

[0018] Based on the reconstruction result and in combination with the preset main ball judgment rule, it is judged whether there is and only one badminton position coordinate that is closest to the three-dimensional coordinate of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image;

[0019] When there is only one badminton position coordinate closest to the three-dimensional coordinate of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image, determine the badminton position coordinate closest to the three-dimensional coordinate of the main ball in the previous frame of the court image as the three-dimensional coordinate of the main ball in the current frame of the court image;

[0020] When there are more than one badminton position coordinates that are closest to the three-dimensional coordinates of the main ball in the previous frame of court image in the reconstruction result of the current frame of court image, it is determined that the main ball does not exist in the current frame of court image.

[0021] In this method, object detection is performed on the court image captured by the image acquisition device at the same time to determine the position coordinates of all the badminton balls in the court image. The reconstruction result is obtained by performing three-dimensional reconstruction on the position coordinates. The main balls are screened out from the reconstruction result for subsequent event judgment. This can greatly reduce the amount of calculation for event perception and further improve the accuracy of event perception, providing users with more accurate event information.

[0022] In an optional implementation, the event information includes hitting time information, hitting speed information, hitting type information, round start information and round end information. Based on the three-dimensional coordinates of the main ball, a preset event perception algorithm is used to determine the event information occurring in the court corresponding to the main ball, including:

[0023] Based on the three-dimensional coordinates of the main requirement, determine whether the moving direction of the main ball in the current time window is the same as the moving direction of the main ball in the previous time window;

[0024] When the moving direction of the main ball in the current time window is different from the moving direction of the main ball in the previous time window, it is determined that a hitting event occurs in the current time window, and the hitting moment information is calculated;

[0025] Based on the hitting moment information, the hitting speed information is calculated using the three-dimensional coordinates of the main ball at the hitting moment and within a preset number of frames after the hitting moment;

[0026] Using the three-dimensional coordinates of the main ball and the hitting speed information, the hitting type information is determined;

[0027] Using the three-dimensional coordinates and hitting speed information of the current main ball, the motion state of the current main ball is identified to obtain the motion state of the current main ball;

[0028] When the current main ball starts from a stationary state and the starting point corresponding to the three-dimensional coordinates of the main ball is located in the serving area of ​​the court, determining that the current main ball is the start of the round;

[0029] When the three-dimensional coordinate corresponding state of the current main ball is the landing state, it is determined that the round of the current main ball has ended.

[0030] In this method, the three-dimensional coordinates of the main ball are used for calculation to identify the hitting moment, hitting speed and hitting type of the main ball on the court at that time, and to determine whether the current main ball corresponds to the serve at the beginning of the round, the badminton touching the net or falling to the ground during the round or at the end of the round, thereby realizing real-time perception of events on the court.

[0031] In an optional embodiment, the method further includes:

[0032] The stadium video stream acquired by the time-synchronized image acquisition device is transferred to the circular queue in the video service module, so that the video service module stores the stadium video stream within the preset time period and plays the stadium video stream sent to the preset time period.

[0033] In this method, the video stream is stored in a circular queue to ensure that the memory usage will not continue to increase, further reducing the computing power consumption of the badminton real-time perception system. By storing and playing the video stream for a period of time in the court, it is convenient for users to grasp the video records in the court in real time.

[0034] In an optional embodiment, the method further includes:

[0035] In response to the user's instruction, the stadium video stream and event information are sent to an external device of the stadium, so that the external device displays the stadium video stream and the event information.

[0036] In this method, by responding to the user's instructions, the stadium's external equipment is used to display the stadium's video stream and the event information, so that the user can grasp the video information on the stadium and the event information occurring on the stadium in real time, achieving a high degree of automation and user interaction, and being compatible with Hawkeye, ball speed measurement and other functions, without the need for dedicated maintenance and operation.

[0037] In a second aspect, the present invention provides a badminton real-time sensing device, the device comprising:

[0038] A video acquisition module is used to acquire a court video stream acquired by an image acquisition device, where the court video stream includes badminton in the court;

[0039] An image conversion module is used to convert the stadium video stream into a stadium image, and store the stadium images with the same frame number in the stadium into a queue in a shared memory;

[0040] An image reconstruction module is used to obtain court images acquired by different image acquisition devices at the same time from a queue in a shared memory, reconstruct the court images at the same time, and obtain the three-dimensional coordinates of a main ball, where the main ball is used to represent a badminton located in the court;

[0041] The event sensing module is used to determine the event information occurring in the court corresponding to the main ball based on the three-dimensional coordinates of the main ball and using a preset event sensing algorithm. The event information is used to characterize the event information occurring at different times on the badminton court.

[0042] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the badminton real-time sensing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the badminton real-time sensing method of the first aspect or any corresponding embodiment thereof.

[0044] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the badminton real-time sensing method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 4 is a flow chart of a badminton real-time sensing method according to an embodiment of the present invention.

[0047] Figure 2 It is a schematic diagram of setting up an image acquisition device according to an embodiment of the present invention.

[0048] Figure 3 It is a schematic diagram of the position of placing Aruco according to an embodiment of the present invention.

[0049] Figure 4 Schematic diagram of the architecture of a badminton real-time perception and interaction system according to an embodiment of the present invention.

[0050] Figure 5 4 is a flow chart of another badminton real-time sensing method according to an embodiment of the present invention.

[0051] Figure 6 4 is a flow chart of another badminton real-time sensing method according to an embodiment of the present invention.

[0052] Figure 7 4 is a structural block diagram of a badminton real-time sensing device according to an embodiment of the present invention.

[0053] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] Among the related technologies, the existing technologies on the market have several technical problems. Existing technical solutions often require many high-speed cameras to capture information on the court, and require high computing power to support the analysis of the content on the court, making the assumptions and operation of the system very costly. At the same time, the existing systems often require a lot of manual participation. These technical problems make it difficult for existing badminton real-time perception and interaction to be applied in daily playing scenarios.

[0056] In order to solve the above problems, a badminton real-time perception method is provided in an embodiment of the present invention, which is used in a computer device. It should be noted that its execution subject can be a badminton real-time perception device, which can be implemented as part or all of the computer device through software, hardware or a combination of software and hardware, wherein the computer device can be a terminal or a client or a server, and the server can be a server or a server cluster composed of multiple servers. The terminal in the embodiment of the present application can be a smart phone, a personal computer, a tablet computer and other intelligent hardware devices. In the following method embodiments, the execution subject is a computer device as an example for explanation.

[0057] The computer device in this embodiment is suitable for sensing events occurring in real time on the court and feeding back to the user. The present invention provides a badminton real-time sensing method, and the video stream in the court can be fully captured by an image acquisition device, the video stream is converted into an image and the image is reconstructed to obtain the three-dimensional coordinates of the badminton, and the events occurring in the court are sensed using the three-dimensional coordinates of the badminton. The video stream acquisition can be realized by using a common image acquisition device set in the court, and the real-time perception of badminton can be realized at a relatively low cost, and the perception and analysis of events on the court can be realized without high computing power. At the same time, manual participation can be greatly reduced, and the perception and analysis of events on the court can be automatically realized. It can also realize the perception and analysis of different courts, so that the perception and analysis of court events can be applied to daily playing scenes, with the potential for large-scale promotion, and greatly improve the user's experience.

[0058] According to an embodiment of the present invention, an embodiment of a badminton real-time sensing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] In this embodiment, a badminton real-time perception method is provided, which can be used in the above-mentioned computer device. Figure 1 is a flow chart of a badminton real-time sensing method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0060] Step S101, obtaining a stadium video stream acquired by an image acquisition device.

[0061] In the embodiment of the present invention, the court video stream includes badminton in the court.

[0062] In one example, the stadium video stream can be captured by ordinary security cameras, pan-tilt monitors, electronic eyes and other image capture devices, and the type of image capture device is not limited in the present invention. Figure 2 is a schematic diagram of the installation of an image acquisition device according to an embodiment of the present invention, such as Figure 2 As shown, select ordinary security cameras and place them outside the four corners of the badminton court, with a height of 3 to 4 meters. Adjust the camera angle so that each camera can capture the entire court.

[0063] Before using the camera for the first time, you should first calibrate the camera's internal parameters. The camera's internal parameters are usually calibrated before leaving the factory, or you can use Zhang Youzheng's calibration method to calibrate it. Use Aruco code No. 1 to 12 to print the plastic board. Figure 3 is a schematic diagram of the position of Aruco placement according to an embodiment of the present invention, such as Figure 3 As shown, place the corresponding board at the specified position, take the upper right corner of the court as the origin of the world coordinate system, the long side as the X-axis, the short side as the Y-axis, and the spatial height as the Z-axis (omitted in the schematic diagram) to establish a world coordinate system, so as to obtain the world coordinate values ​​corresponding to the 4 corner points in each Aruco according to the scale of the printed board (the board is on the ground, so the z value of each point is 0). After this step is calibrated, the internal parameters are recorded for repeated calls in subsequent steps. Every time the system is started, in order to prevent the camera from being moved by people, the external parameters of the camera need to be calibrated. Use a key point detection algorithm (such as a key point detection model or other computer vision algorithm) to detect the intersection of the badminton court ground line, and correspond it to the real coordinate system, so that the external parameters of each camera can be calculated.

[0064] Step S102, converting the stadium video stream into a stadium image, and storing the stadium images with the same frame number in the stadium into a queue of the shared memory.

[0065] In one example, the video stream processing module is used to obtain the video stream from the camera, and the camera input of multiple courts can be processed simultaneously. The video stream processing module uses GPU hard decoding to convert the video stream into RGB format images. After receiving four images with the same frame number of the same court, the images are stored in a shared memory queue.

[0066] Step S103, obtaining the stadium images acquired at the same time by different image acquisition devices from the queue of the shared memory, reconstructing the stadium images at the same time, and obtaining the three-dimensional coordinates of the main ball.

[0067] In an embodiment of the present invention, the main ball is used to represent a badminton shuttlecock located in a court.

[0068] In one example, the three-dimensional coordinates of the main ball are reconstructed through the following steps: (1) Take out the pictures taken by four cameras at the same time on the same field from the queue in the shared memory. (2) Use the object detection algorithm to detect the badminton in the picture. (3) Use the 3D reconstruction algorithm to reconstruct the detected badminton into three-dimensional coordinates. (4) Filter out the balls outside the field and the stationary balls from the reconstructed balls, and select the ball that is closest to the main ball (if any) of the previous frame from the remaining balls as the main ball of this frame. If the selection is not unique, there is no main ball in this frame. The main ball refers to the ball that the players on the field are hitting.

[0069] Step S104, based on the three-dimensional coordinates of the main ball, using a preset event perception algorithm, determine the event information occurring in the court corresponding to the main ball.

[0070] In the embodiment of the present invention, the event information is used to represent the event information of the badminton occurring at different times in the court.

[0071] In one example, the event perception algorithm receives the main ball coordinates reconstructed by the badminton real-time reconstruction module, and uses the algorithm to determine the events currently occurring on the court in real time. Events include the moment of hitting the ball, the speed of the ball, the type of hitting, the start of a round (serve), the end of a round (ball touching the net or falling to the ground), etc.

[0072] In one implementation scenario, Figure 4 is a schematic diagram of the architecture of a badminton real-time perception and interaction system according to an embodiment of the present invention. Figure 4 As shown in the figure, the badminton real-time perception and interaction system collects video streams through the camera and outputs the video streams to the large screen of the stadium; the video stream processing module obtains the video stream from the camera, and interacts with the badminton real-time reconstruction module through the shared memory for video stream data; the badminton real-time reconstruction module transmits the reconstructed three-dimensional coordinates of the main ball to the event perception module, transfers the video stream to the video service module, and sends the three-dimensional coordinates of the main ball to the interaction module located in the cloud; the event perception module perceives the event information occurring on the court, issues video-related instructions to the video service module, and outputs the event information to other court peripherals (such as LED dot matrix display, speakers, etc.) and cloud servers (interaction modules, etc.); the video service module sends the video stream to the large screen of the stadium, and interacts with the interaction module for video upload / download, thereby realizing real-time perception and interaction of badminton court events.

[0073] The badminton real-time perception method provided in this embodiment can fully capture the video stream in the court through the image acquisition device, convert the video stream into an image and reconstruct the image to obtain the three-dimensional coordinates of the badminton, and use the three-dimensional coordinates of the badminton to perceive the events occurring in the court. The video stream acquisition can be realized by using the ordinary image acquisition device set in the court, and the real-time perception of badminton can be realized at a relatively low cost. The perception and analysis of events on the court can be realized without high computing power. At the same time, manual participation can be greatly reduced, and the perception and analysis of events on the court can be realized automatically. It can also realize the perception and analysis of different courts, so that the perception and analysis of court events can be applied to daily playing scenes, which has the potential for large-scale promotion and greatly improves the user experience.

[0074] In this embodiment, a badminton real-time perception method is provided, which can be used in the above-mentioned computer device. Figure 5 is a flow chart of another badminton real-time sensing method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0075] Step S501, obtain the stadium video stream acquired by the image acquisition device. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0076] Step S502, converting the stadium video stream into a stadium image, and storing the stadium images with the same frame number in the stadium into a queue of the shared memory.

[0077] In an optional implementation, converting a stadium video stream into a stadium image includes: time synchronizing an image acquisition device to obtain a time-synchronized image acquisition device; using the time-synchronized image acquisition device to acquire a stadium video stream, and converting the stadium video stream into a stadium image through GPU hard decoding.

[0078] In one example, in order to obtain accurate badminton reconstruction results, the camera needs to be time synchronized before the system starts. Using the edge server in the same LAN segment as the NTP server, and using the NTP protocol to correct the time of the clock on the camera, it can be ensured that there is no time difference between the four cameras in the same court, or the time difference is within a certain allowable range, such as within 200 milliseconds. The specific value of the time difference is not limited in the present invention. After the time synchronization process is completed, the pictures sent by the four cameras with the same timestamp are pictures of the same moment in the real world. Using the video stream processing module, GPU hard decoding is used to convert the video stream into RGB format pictures. After receiving four pictures with the same frame number in the same venue, the pictures are stored in a queue of a shared memory.

[0079] In this method, in order to obtain accurate badminton reconstruction results, the image acquisition devices are time synchronized so that there is no time difference between the image acquisition devices, or the time difference is within an allowable range, which facilitates the subsequent three-dimensional reconstruction of the court image obtained by converting the video stream captured by the time-synchronized image acquisition device.

[0080] Step S503, obtaining the stadium images acquired at the same time by different image acquisition devices from the queue of the shared memory, reconstructing the stadium images at the same time, and obtaining the three-dimensional coordinates of the main ball.

[0081] Specifically, the above step S503 includes:

[0082] Step S5031, using an object detection algorithm to detect the court image at the same time, and obtain the position coordinates of all badminton balls in each frame of the court image.

[0083] In an example, this step is explained in detail.

[0084] Step S5032, performing three-dimensional reconstruction on the badminton position coordinates to obtain reconstruction results corresponding to the badminton position coordinates.

[0085] Step S5033, based on the reconstruction result and in combination with the preset main ball judgment rule, determine whether there is and only one badminton position coordinate that is closest to the three-dimensional coordinates of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image.

[0086] Step S5034, when there is only one badminton position coordinate that is closest to the three-dimensional coordinates of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image, the badminton position coordinate that is closest to the three-dimensional coordinates of the main ball in the previous frame of the court image is determined as the three-dimensional coordinates of the main ball in the current frame of the court image.

[0087] Step S5035, when there are more than one badminton position coordinates that are closest to the three-dimensional coordinates of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image, it is determined that the main ball does not exist in the current frame of the court image.

[0088] In one example, a badminton real-time reconstruction module is used to reconstruct the court image at the same time, which includes the following steps: (1) Take out the pictures taken by four cameras at the same time in the same venue from the queue of the shared memory. (2) Use the object detection algorithm to detect the position coordinates of the badminton in the picture. The object detection algorithm is a deep learning model, and the model structure can use the structure of Yolov5, etc. The specific model is not limited in the present invention. The input of the object detection algorithm is a picture, and the output is the position coordinates of all badmintons in the picture. (3) The picture of a fixed camera among the four cameras is transferred to a circular queue in the video service module. This circular queue will store the court video data in the past period of time. The reason for using the circular queue is to ensure that the memory usage will not continue to increase. (4) Use the 3D reconstruction algorithm to reconstruct the detected badminton into three-dimensional coordinates. The simple principle of the 3D reconstruction algorithm is: each ball detected in the picture corresponds to a ray emitted from the camera position in space. If two cameras detect the same ball, then the closest point of the two rays can be calculated to obtain the spatial coordinates of the badminton. For details, please refer to the supplementary information. (5) Filter out the balls outside the court and the motionless balls from the reconstructed balls, and select the ball closest to the main ball (if any) in the previous frame as the main ball of this frame. If the selection is not unique, there is no main ball in this frame. The main ball refers to the ball that the players on the court are playing. The coordinates of the main ball and the current frame number will be transmitted to the event perception module and the interaction module of the cloud server.

[0089] Specifically, the screening conditions for selecting the main ball include: 1. The main ball should be within the boundaries of the court; 2. The main ball should not be stationary (the position does not change for 20 frames); 3. If there is a main ball in the previous frame, select the ball closest to the main ball of the previous frame; 4. If there are multiple candidates that cannot be determined, there is no main ball in this frame.

[0090] In this method, object detection is performed on the court image captured by the image acquisition device at the same time to determine the position coordinates of all the badminton balls in the court image. The reconstruction result is obtained by performing three-dimensional reconstruction on the position coordinates. The main balls are screened out from the reconstruction result for subsequent event judgment. This can greatly reduce the amount of calculation for event perception and further improve the accuracy of event perception, providing users with more accurate event information.

[0091] Step S504: Based on the three-dimensional coordinates of the main ball, the event information occurring in the court corresponding to the main ball is determined using a preset event perception algorithm. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0092] The badminton real-time perception method provided in this embodiment, in order to obtain accurate badminton reconstruction results, synchronizes the image acquisition devices in time so that there is no time difference between the image acquisition devices, or the time difference is within an allowable range, so as to facilitate the subsequent three-dimensional reconstruction of the court image obtained by converting the video stream acquired by the time-synchronized image acquisition device. Object detection is performed on the court image acquired by the image acquisition device at the same time, the position coordinates of all badmintons in the court image are determined, and the reconstruction result is obtained by three-dimensional reconstruction of the position coordinates. The main ball is selected from the reconstruction result for subsequent event judgment, which can greatly reduce the calculation amount of event perception, and at the same time can further improve the accuracy of event perception, and provide users with more accurate event information.

[0093] In this embodiment, a badminton real-time perception method is provided, which can be used in the above-mentioned computer device. Figure 6 is a flow chart of another badminton real-time sensing method according to an embodiment of the present invention, such as Figure 6 As shown, the process includes the following steps:

[0094] Step S601, obtain the stadium video stream acquired by the image acquisition device. Figure 5 Step S501 of the illustrated embodiment will not be described in detail here.

[0095] Step S602: Convert the stadium video stream into stadium images, and store the stadium images with the same frame number in the stadium into a queue in the shared memory. Figure 5 Step S502 of the illustrated embodiment will not be described in detail here.

[0096] Step S603, obtain the stadium images acquired by different image acquisition devices at the same time from the queue of the shared memory, reconstruct the stadium images at the same time, and obtain the three-dimensional coordinates of the main ball. Figure 5 Step S503 of the illustrated embodiment will not be described in detail here.

[0097] Step S604, based on the three-dimensional coordinates of the main ball, using a preset event perception algorithm, determine the event information occurring in the court corresponding to the main ball.

[0098] Specifically, the event information includes batting time information, batting speed information, batting type information, round start information, and round end information. The above step S604 includes:

[0099] Step S6041, based on the three-dimensional coordinates of the main requirement, determine whether the moving direction of the main ball in the current time window is the same as the moving direction of the main ball in the previous time window.

[0100] Step S6042, when the moving direction of the main ball in the current time window is different from the moving direction of the main ball in the previous time window, it is determined that a hitting event occurs in the current time window, and the hitting moment information is calculated.

[0101] Step S6043, based on the hitting moment information, the hitting speed information is calculated using the three-dimensional coordinates of the main ball within the hitting moment and the preset number of frames after the hitting moment.

[0102] Step S6044, using the three-dimensional coordinates of the main ball and the hitting speed information, determine the hitting type information.

[0103] Step S6045, using the three-dimensional coordinates and hitting speed information of the current main ball, the motion state of the current main ball is identified to obtain the motion state of the current main ball.

[0104] Step S6046, when the current main ball starts from a stationary state and the starting point corresponding to the three-dimensional coordinates of the main ball is located in the serving area of ​​the court, it is determined that the current main ball is the start of the round.

[0105] Step S6047, when the three-dimensional coordinate corresponding state of the current main ball is the landing state, it is determined that the round of the current main ball is over.

[0106] In one example, the event information occurring in the court corresponding to the main ball is determined by the event perception module, which includes two parts: an event perception algorithm and an interface with the interaction module. The event perception algorithm receives the coordinates of the main ball reconstructed by the badminton real-time reconstruction module, and uses the algorithm to determine the events occurring on the current court in real time. The events include the time of hitting the ball, the speed of hitting the ball, the type of hitting the ball, the start of the round (serving), the end of the round (the ball touches the net or falls to the ground), etc.

[0107] Specifically, the event perception algorithm example is as follows: after receiving the new main ball coordinates, determine whether the moving direction of the badminton in the most recent time window is the same as that in the previous time window. If a change occurs, it means that a hitting event has occurred here. For example, in a time window, the x coordinates of the first half of the time and the x coordinates of the second half of the time are both greater than or less than the x coordinates of the middle frame, then it can be determined that the ball has turned back. The hitting speed is calculated using the displacement and time difference of the badminton at the time of hitting the ball and the next few frames, and it is determined that the ball has turned back in the Nth frame. The coordinates of the Nth frame and the N+1th frame are used to calculate the displacement and time difference, and the speed is obtained by dividing. The flying height, flying speed, flying distance and other characteristics of the ball are used to determine the hitting category. For example, if the flying distance and flying height of the ball reach a certain threshold, it is a high and far ball; if the flying height, flying time, and falling angle of the ball reach a certain threshold, it is a pick-up ball, which is not limited in the present invention. If the badminton starts from a state of approximate stillness and the starting point is located in the serving area, it is judged as a serve; if the badminton lands, it is judged as the end of the round.

[0108] The event perception module uses the event perception algorithm and the interface with the interaction module to send information or instructions to the video service module and the cloud interaction module based on the events output by the event perception algorithm. For example, after determining that the ball has landed, the video service module is notified to capture the video after the last serve, or after calculating the ball speed, the speaker peripheral is notified to broadcast the ball speed, etc. After receiving the instruction, the video service module synthesizes the corresponding video and sends it to the local interaction module such as the stadium's large screen or the cloud interaction module. The cloud interaction module can provide the received video and other information to the user. The event perception algorithm and the interaction interface should be flexibly configured according to usage requirements, and have no coupling relationship with the system, and are not limited in the present invention.

[0109] In this method, the three-dimensional coordinates of the main ball are used for calculation to identify the hitting moment, hitting speed and hitting type of the main ball on the court at that time, and to determine whether the current main ball corresponds to the serve at the beginning of the round, the badminton touching the net or falling to the ground during the round or at the end of the round, thereby realizing real-time perception of events on the court.

[0110] Step S605, transferring the stadium video stream acquired by the time-synchronized image acquisition device to a circular queue in the video service module, so that the video service module stores the stadium video stream within a preset time period and plays the stadium video stream sent to the preset time period.

[0111] In one example, the badminton real-time reconstruction module transfers the image of a fixed camera among the four cameras into a circular queue in the video service module, which stores the video data of the court in the past period of time. The reason for using a circular queue is to ensure that the memory usage does not continue to increase. The video service module uses a circular queue to accept the images transferred from the badminton real-time reconstruction module to record the video of a court in the past period of time. The video service module receives instructions from the event perception module, and synthesizes the recorded content into a video according to a certain resolution, clarity, frame rate and duration as required, and then sends it to the large screen of the venue or uploads it to the cloud interaction module. The video service module can also download some videos generated from the cloud from the cloud, and then send them to the large screen of the venue for playback.

[0112] In this method, the video stream is stored in a circular queue to ensure that the memory usage will not continue to increase, further reducing the computing power consumption of the badminton real-time perception system. By storing and playing the video stream for a period of time in the court, it is convenient for users to grasp the video records in the court in real time.

[0113] Step S606, in response to the user's instruction, the stadium video stream and the event information are sent to an external device of the stadium, so that the external device displays the stadium video stream and the event information.

[0114] In one example, the interaction module includes a local interaction module and a cloud interaction module. The local interaction module includes peripherals such as a large screen in the venue, and the types are not limited to TV screens, projectors, dot matrix screens, speakers, lighting controls, etc. The cloud interaction module includes a cloud server and a mobile app (including mini-programs). The local interaction module receives the output of the event perception module and the video service module, and provides users in the venue with functions such as wonderful playback and ball speed broadcasting. The server of the cloud interaction module receives videos, main ball coordinates, instructions and other information uploaded by the edge server, and provides users with corresponding information based on requests issued by users on the mobile phone, including ball speed statistics, Hawkeye Challenge and other functions.

[0115] In this method, by responding to the user's instructions, the stadium's external equipment is used to display the stadium's video stream and the event information, so that the user can grasp the video information on the stadium and the event information occurring on the stadium in real time, achieving a high degree of automation and user interaction, and being compatible with Hawkeye, ball speed measurement and other functions, without the need for dedicated maintenance and operation.

[0116] The badminton real-time perception method provided in this embodiment uses the three-dimensional coordinates of the main ball for calculation, identifies the hitting time, hitting speed and hitting type corresponding to the main ball in the court at that time, determines whether the current main ball corresponds to the serve at the beginning of the round, the badminton touching the net or falling during the round or at the end of the round, and realizes the real-time perception of events in the court. The video stream is stored in a circular queue to ensure that the memory usage will not continue to increase, further reducing the computing power consumption of the badminton real-time perception system. By storing and playing the video stream within a period of time in the court, it is convenient for users to grasp the video records in the court in real time. By responding to the user's instructions, the court video stream and the event information are displayed using the external device of the court, which is convenient for users to grasp the video information on the court and the event information occurring on the court in real time, realizing a high degree of automation and user interaction, compatible with functions such as Hawkeye and ball speed measurement, and no special person maintenance and operation is required.

[0117] In the present embodiment, a badminton real-time sensing device is also provided, and the device is used to realize the above-described embodiment and the preferred embodiment, and the description thereof will not be repeated. As used below, the term "module" can realize the combination of software and / or hardware of the predetermined function. Although the device described in the following embodiments is preferably realized with software, the realization of hardware, or the combination of software and hardware is also possible and conceived.

[0118] This embodiment provides a badminton real-time sensing device, such as Figure 7 As shown, including:

[0119] The video acquisition module 701 is used to acquire the court video stream acquired by the image acquisition device, and the court video stream includes the badminton in the court. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0120] The image conversion module 702 is used to convert the stadium video stream into a stadium image, and store the stadium images with the same frame number in the stadium into a queue in the shared memory. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0121] The image reconstruction module 703 is used to obtain the court images acquired by different image acquisition devices at the same time from the queue of the shared memory, reconstruct the court images at the same time, and obtain the three-dimensional coordinates of the main ball. The main ball is used to represent the badminton located in the court. For details, please refer to Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0122] The event sensing module 704 is used to determine the event information occurring in the court corresponding to the main ball based on the three-dimensional coordinates of the main ball and using a preset event sensing algorithm. The event information is used to characterize the event information occurring at different times in the court for badminton. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0123] In some optional implementations, the image conversion module 702 includes:

[0124] The device time synchronization unit is used to perform time synchronization on the image acquisition device to obtain a time-synchronized image acquisition device.

[0125] The image and video conversion unit is used to acquire a stadium video stream using a time-synchronized image acquisition device, and convert the stadium video stream into a stadium image through GPU hard decoding.

[0126] In some optional implementations, the image reconstruction module 703 includes:

[0127] The object detection unit is used to detect the court image at the same time using an object detection algorithm to obtain the position coordinates of all badminton balls in each frame of the court image.

[0128] The three-dimensional reconstruction unit is used to perform three-dimensional reconstruction on the badminton position coordinates to obtain reconstruction results corresponding to the badminton position coordinates.

[0129] The main ball judgment unit is used to judge whether there is and only one badminton position coordinate that is closest to the three-dimensional coordinates of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image based on the reconstruction result and the preset main ball judgment rule.

[0130] A main ball determination unit is used to determine the badminton position coordinates closest to the three-dimensional coordinates of the main ball in the previous frame of the court image as the three-dimensional coordinates of the main ball in the current frame of the court image when there is only one badminton position coordinates closest to the three-dimensional coordinates of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image.

[0131] The main ball does not exist unit determines that the main ball does not exist in the current frame court image when there are more than one badminton position coordinates that are closest to the three-dimensional coordinates of the main ball in the previous frame court image in the reconstruction result of the current frame court image.

[0132] In some optional implementations, the event sensing module 704 includes:

[0133] The moving direction judgment unit is used to judge whether the moving direction of the main ball in the current time window is the same as the moving direction of the main ball in the previous time window based on the three-dimensional coordinates of the main requirement.

[0134] The hitting moment calculation unit is used to determine that a hitting event occurs in the current time window and calculate the hitting moment information when the moving direction of the main ball in the current time window is different from the moving direction of the main ball in the previous time window.

[0135] The batting speed calculation unit is used to calculate the batting speed information based on the batting time information and the three-dimensional coordinates of the main ball within the preset frame number after the batting time.

[0136] The shot type determination unit is used to determine the shot type information using the three-dimensional coordinates of the main ball and the shot speed information.

[0137] The motion state recognition unit is used to recognize the motion state of the current main ball by using the three-dimensional coordinates and hitting speed information of the current main ball, and recognize the motion state of the current main ball.

[0138] The round start determination unit is used to determine that the current main ball is the start of the round when the current main ball starts from a stationary state and the starting point corresponding to the three-dimensional coordinates of the main ball is located in the serving area of ​​the court.

[0139] The round end determination unit is used to determine that the current main ball is at the end of the round when the three-dimensional coordinate corresponding state of the current main ball is the landing state.

[0140] In some optional implementations, the badminton real-time sensing device further includes:

[0141] The video service unit is used to transfer the stadium video stream acquired by the time-synchronized image acquisition device to the circular queue in the video service module, so that the video service module stores the stadium video stream within a preset time period and plays the stadium video stream sent to the preset time period.

[0142] In some optional implementations, the badminton real-time sensing device further includes:

[0143] The interactive unit is used to send the stadium video stream and event information to an external device of the stadium in response to a user's instruction, so that the external device can display the stadium video stream and the event information.

[0144] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0145] The badminton real-time sensing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0146] The embodiment of the present invention also provides a computer device having the above Figure 7 The badminton real-time sensing device shown.

[0147] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0148] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0149] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0150] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0151] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0152] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0153] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0154] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0155] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0156] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A badminton real-time perception method, characterized in that: The method comprises: Acquire a court video stream acquired by an image acquisition device, wherein the court video stream includes badminton in the court; Converting the stadium video stream into stadium images, and storing the stadium images with the same frame numbers in the stadium into a queue of a shared memory; Obtaining court images acquired at the same time by different image acquisition devices from the queue of the shared memory, reconstructing the court images at the same time, and obtaining the three-dimensional coordinates of a main ball, wherein the main ball is used to represent a badminton located in the court; Based on the three-dimensional coordinates of the main ball, a preset event perception algorithm is used to determine event information occurring in the court corresponding to the main ball, and the event information is used to characterize event information occurring in the badminton court at different times.

2. The method according to claim 1, characterized in that The converting the stadium video stream into a stadium image comprises: Performing time synchronization on the image acquisition device to obtain a time-synchronized image acquisition device; The stadium video stream is acquired by using the time-synchronized image acquisition device, and the stadium video stream is converted into a stadium image by GPU hard decoding.

3. The method according to claim 1, characterized in that The reconstructing the stadium image at the same time includes: Using an object detection algorithm to detect the court image at the same time, and obtain the position coordinates of all badminton balls in each frame of the court image; Performing three-dimensional reconstruction on the badminton position coordinates to obtain a reconstruction result corresponding to the badminton position coordinates; Based on the reconstruction result and in combination with a preset main ball judgment rule, it is determined whether there is and only exists one badminton position coordinate that is closest to the three-dimensional coordinate of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image; When there is only one badminton position coordinate closest to the three-dimensional coordinate of the main ball in the previous frame of the court image in the reconstruction result of the current frame of the court image, determine the badminton position coordinate closest to the three-dimensional coordinate of the main ball in the previous frame of the court image as the three-dimensional coordinate of the main ball in the current frame of the court image; When there are more than one badminton position coordinates that are closest to the three-dimensional coordinates of the main ball in the previous frame of court image in the reconstruction result of the current frame of court image, it is determined that the main ball does not exist in the current frame of court image.

4. The method according to claim 3, characterized in that The event information includes hitting time information, hitting speed information, hitting type information, round start information and round end information. The event information occurring in the court corresponding to the main ball is determined based on the three-dimensional coordinates of the main ball using a preset event perception algorithm, including: Based on the three-dimensional coordinates of the main requirement, determining whether the moving direction of the main ball in the current time window is the same as the moving direction of the main ball in the previous time window; When the moving direction of the main ball in the current time window is different from the moving direction of the main ball in the previous time window, it is determined that a hitting event occurs in the current time window, and the hitting moment information is calculated; Based on the hitting moment information, the hitting speed information is calculated using the hitting moment and the three-dimensional coordinates of the main ball within a preset number of frames after the hitting moment; Determine the ball hitting type information using the three-dimensional coordinates of the main ball and the ball hitting speed information; Using the three-dimensional coordinates of the current main ball and the hitting speed information, the motion state of the current main ball is identified to obtain the motion state of the current main ball; When the current main ball starts from a stationary state and the starting point corresponding to the three-dimensional coordinates of the main ball is located in the serving area of ​​the court, determining that the current main ball is the start of a round; When the three-dimensional coordinate corresponding state of the current main ball is the landing state, it is determined that the round of the current main ball has ended.

5. The method according to claim 2, characterized in that: The method further comprises: The stadium video stream acquired by the time-synchronized image acquisition device is transferred to a circular queue in the video service module, so that the video service module stores the stadium video stream within a preset time period and plays the stadium video stream sent to the preset time period.

6. The method according to claim 5, characterized in that The method further comprises: In response to a user's instruction, the stadium video stream and the event information are sent to an external device of the stadium, so that the external device displays the stadium video stream and the event information.

7. A badminton real-time sensing device, characterized in that: The device comprises: A video acquisition module, used to acquire a court video stream acquired by an image acquisition device, wherein the court video stream includes badminton in the court; An image conversion module, used for converting the stadium video stream into a stadium image, and storing the stadium images with the same frame number in the stadium into a queue of a shared memory; An image reconstruction module, used to obtain court images acquired at the same time by different image acquisition devices from the queue of the shared memory, reconstruct the court images at the same time, and obtain the three-dimensional coordinates of the main ball, where the main ball is used to represent the badminton located in the court; The event sensing module is used to determine the event information occurring in the court corresponding to the main ball based on the three-dimensional coordinates of the main ball and using a preset event sensing algorithm, wherein the event information is used to characterize the event information occurring at different times of the badminton in the court.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the badminton real-time sensing method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the badminton real-time sensing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the badminton real-time sensing method according to any one of claims 1 to 6.

Images