Live-action auxiliary penalty method and system

Abstract

The invention provides a live-action auxiliary penalty method and system. The method comprises the following steps: acquiring video data of a tennis court and tennis balls and human bodies in the tennis court acquired by a camera device in real time; monitoring a tennis ball landing event in the video data, and recording an occurrence time point of the tennis ball landing event; acquiring a plurality of target image frames in the video data within a preset duration before and after the occurrence time point; tennis ball contours of the sports tennis balls are extracted from the multiple target image frames; drawing a tennis ball contour extracted from the plurality of target image frames on a site live-action picture so as to form a live-action dynamic video and / or a live-action static image which correspond to the tennis ball landing event and comprise a tennis ball motion trail; and performing auxiliary penalty according to the live-action dynamic video and / or the live-action static image to generate a penalty result, wherein the penalty result comprises a judgment result of whether the tennis ball is out of the bound and the live-action dynamic video and / or the live-action static image. According to the invention, the accuracy and authenticity of motion analysis by using a common camera are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sports equipment, and in particular to a real-scene auxiliary penalty judgment method and system. Background Art

[0002] Hawkeye is also known as the instant replay system. It is a powerful auxiliary tool for the accuracy of referee's penalty. The principle of this technology is not complicated, but it is very precise. This system consists of 8 or 10 high-speed cameras, four computers and a large screen. This system is divided into the following steps. First, the three-dimensional space in the competition venue is divided into measurement units calculated in millimeters with the help of computer calculations; then, the basic data of the flight trajectory of the tennis ball is captured from different angles at the same time using high-speed cameras; then, through computer calculations, these data are fitted with curves in the three-dimensional coordinate system to generate a three-dimensional fitting picture; finally, using instant imaging technology, the movement route and landing point of the tennis ball are clearly presented on the large screen. From data collection to result presentation, the time consumed in this process does not exceed 10 seconds.

[0003] In summary, the professional Hawkeye system uses fitted images during playback, and requires the collaboration of multiple high-speed cameras to capture every frame of the tennis ball, which is expensive. If ordinary cameras are used, the frame rate is not enough to accurately determine the landing point of the ball, and the accuracy of the professional Hawkeye system cannot be achieved. Therefore, the professional Hawkeye system is difficult to be popularized in the field of national fitness. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a real-scene assisted penalty judgment method and system that solves the above technical problems or at least partially solves the above technical problems.

[0005] One aspect of the present invention provides a real-scene assisted penalty judgment method, the method comprising:

[0006] Acquire video data collected by a camera device in real time on a tennis court and tennis balls and human bodies located in the tennis court;

[0007] Monitoring a tennis ball falling event in the video data, and recording a time point at which the tennis ball falling event occurs in the video data;

[0008] Acquire multiple target image frames in the video data that are within a preset time length before and after the occurrence time point;

[0009] extracting tennis ball contours of the moving tennis ball from the plurality of target image frames respectively;

[0010] Drawing the tennis ball outline extracted from the plurality of target image frames on a preset field real scene image to form a real-scene dynamic video and / or a real-scene static image including a tennis ball movement trajectory corresponding to the tennis ball landing event;

[0011] An auxiliary penalty is performed based on the real-scene dynamic video and / or the real-scene static image and a penalty result is generated, where the penalty result includes a determination result of whether the tennis ball is out of bounds and the real-scene dynamic video and / or the real-scene static image.

[0012] Furthermore, the method further comprises: pre-generating a real scene map of the site;

[0013] Pre-generate site real-life pictures, including:

[0014] Use Gaussian mixture model to construct the background model of the real scene of the tennis court;

[0015] The preset scene scene image is pixel-matched with the background model, the foreground image in the scene scene image is detected and the foreground image is removed from the tennis court scene image, so as to obtain a field scene image consistent with the actual scene of the tennis court.

[0016] Furthermore, extracting the tennis ball contours of the moving tennis ball from the plurality of target image frames respectively includes:

[0017] Extracting the initial mask image corresponding to each target image frame, which only describes the pixels of the moving objects in the current image;

[0018] The moving objects in each initial mask image are enhanced by morphological operations to obtain a standard mask image with smooth contours of the moving objects corresponding to each target image frame.

[0019] The tennis ball contour is detected on the standard mask image of each target image frame to obtain a target mask image containing only the tennis ball contour corresponding to each target image frame.

[0020] Furthermore, extracting an initial mask image corresponding to each target image frame and describing only pixels of moving objects in the current image includes:

[0021] Performing a differential operation on each target image frame and the site real scene image to obtain a global difference image corresponding to each target image frame;

[0022] Performing a differential operation on each target image frame and a previous image frame adjacent to the current target image frame to obtain a local difference image corresponding to each target image frame;

[0023] The global difference image and the local difference image corresponding to each target image frame are integrated to obtain an initial mask image corresponding to the current target image frame that only describes the pixels of the moving object.

[0024] Furthermore, the edge enhancement processing of the moving object in each initial mask image by morphological operation includes:

[0025] The moving objects in each initial mask image are expanded outward through the first dilation operation to enhance the visibility and edges of the moving objects, the moving objects are eroded through the opening operation to remove noise in the image, and the moving objects are expanded outward through the second dilation operation to restore the size of the moving objects.

[0026] Furthermore, the detecting of tennis ball contours on the standard mask image of each target image frame includes:

[0027] Track the boundaries of the moving objects in the standard mask image of each target image frame to find the closed area corresponding to the moving objects, store the contours corresponding to the closed area and / or the pixels within the contours, and obtain the contour information of all the moving objects in the image;

[0028] The contour information of each moving object is identified according to at least one preset tennis feature recognition rule, and the tennis contour in the standard mask image of each target image frame is detected.

[0029] Furthermore, the tennis feature recognition rules include:

[0030] Detecting whether the contour area corresponding to each moving object meets a preset area threshold range according to the contour information of the moving object; and / or

[0031] Detecting whether the contour shape corresponding to each moving object meets the preset shape parameter requirements according to the contour information of the moving object; and / or

[0032] A vector relationship detection is performed on the contour information of the moving object and the tennis ball contour detected in at least one adjacent continuous target image frame to determine whether the vector relationship of the continuous multiple contours conforms to the motion trajectory.

[0033] Furthermore, the step of tracking the boundary of the moving object in the standard mask image to find the closed area corresponding to the moving object includes:

[0034] Starting from each pixel in the standard mask image, chain encoding is used to traverse along the boundary of the moving object to find adjacent contour pixels until a closed area corresponding to the moving object is found.

[0035] Further, drawing the tennis ball outline extracted from the plurality of target image frames on a preset field real scene image includes:

[0036] Performing bitwise AND operations on the standard mask images of each target image frame and the real scene image of the venue, respectively, so as to merge the outline of the tennis ball in each standard mask image into the real scene image of the venue, thereby obtaining a continuous real scene image sequence including only the tennis ball, and generating a real scene dynamic video including the movement trajectory of the tennis ball corresponding to the tennis ball landing event according to the real scene image sequence; and / or

[0037] The standard mask images of each target image frame are bitwise ANDed with the same field real scene image to merge the tennis ball outlines in each standard mask image into a field real scene image, thereby obtaining a real-scene static image including the tennis ball's motion trajectory corresponding to the tennis ball landing event.

[0038] Another aspect of the present invention provides a real-scene assisted penalty judgment system, the system comprising:

[0039] Cloud processing system, client device, central control machine, display device and at least one camera device;

[0040] A camera device is used to shoot the tennis court and the tennis balls and human bodies in the tennis court to collect live images in real time;

[0041] The central control machine is used to store the video stream collected by the camera device;

[0042] The client device is used to trigger an auxiliary penalty event according to a user request and send the auxiliary penalty request to a cloud processing system;

[0043] The cloud processing system is used to send a penalty processing command to the central control machine when receiving an auxiliary penalty request sent by the client device;

[0044] The central control machine is further used to execute the above-mentioned real-scene assisted penalty judgment method upon receiving the penalty processing command sent by the cloud processing system and send the penalty result to the cloud processing system;

[0045] The cloud processing system is also used to push the penalty results to the on-site display device and / or client device for display.

[0046] The real-scene assisted penalty judgment method and system provided by the embodiment of the present invention captures video images and extracts the movement trajectory of the tennis ball through an ordinary camera device, accurately depicts the position information of the tennis ball before and after it lands on the ground into the real-scene map of the venue, and finally generates a real-scene dynamic video and / or a real-scene static image containing the movement trajectory of the tennis ball, effectively improving the accuracy and authenticity of motion analysis, and can assist in determining the exact position of the tennis ball at the moment of landing and make penalties, thereby solving the technical problems that the existing Hawkeye system including high-speed cameras is expensive and difficult to be popularized in the field of national fitness, and that the existing use of ordinary cameras to assist in penalty judgment cannot accurately determine the landing position of the ball due to insufficient frame rate, and cannot achieve the accuracy of a professional Hawkeye system.

[0047] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0049] Figure 1 Flow chart of the real-scene assisted penalty judgment method according to an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a site real scene diagram proposed in an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of an initial mask image proposed in an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a standard mask image proposed in an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a target mask image proposed in an embodiment of the present invention;

[0054] Figure 6 It is a schematic diagram of a real-scene static image proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0056] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0057] The embodiment of the present invention provides a real-scene assisted penalty judgment method, such as Figure 1 As shown, the real-scene assisted penalty judgment method proposed by the present invention comprises the following steps:

[0058] S1. Obtain video data of a tennis court and a tennis ball and a human body in the tennis court collected in real time by a camera device.

[0059] The real-scene assisted penalty method of this embodiment is applied to a real-scene assisted penalty system, which includes a cloud processing system, a client device and an on-site device, and the on-site device includes a central control machine, a display device, at least one camera device and some network auxiliary facilities. Among them, the display device can be implemented with a large screen. The camera device can be implemented with a camera, and specifically, an ordinary camera or a high-speed camera can be used, which is not limited by the present invention. If the camera includes at least two channels, it can be installed on both sides of the court. If there is only one camera, it can be installed on only one side of the court. The real-scene assisted penalty method of this embodiment is executed by the central control machine.

[0060] S2. Monitor the tennis ball falling event in the video data, and record the time point when the tennis ball falling event occurs in the video data.

[0061] S3, obtaining multiple target image frames in the video data that are located within a preset time length before and after the occurrence time point.

[0062] S4. Extracting tennis ball contours of the moving tennis ball from the multiple target image frames respectively.

[0063] S5. Drawing the tennis ball outline extracted from the plurality of target image frames on a preset field real scene image to form a real-scene dynamic video and / or a real-scene static image including a tennis ball movement trajectory corresponding to the tennis ball landing event.

[0064] S6. Assisting in penalty judgment according to the real-scene dynamic video and / or the real-scene static image and generating a penalty judgment result, the penalty judgment result including a judgment result of whether the tennis ball is out of bounds and the real-scene dynamic video and / or the real-scene static image. In this embodiment, the penalty judgment result includes a direct judgment result of whether the tennis ball is out of bounds: in or out, and an indirect result: the real-scene dynamic video and / or the real-scene static image when the ball landed most recently.

[0065] In this implementation, the central control machine can determine the occurrence of the tennis ball falling to the ground through real-time calculation. Specifically, the identification of the tennis ball falling event adopts a video analysis method, and uses an efficient time series neural network (such as LSTM or GRU) based on a preset data set to dynamically learn the trajectory of the tennis ball moving in the venue, thereby capturing the time and spatial position information of the landing point to realize the monitoring of the tennis ball falling event of the real-time collected live video data, so that the accuracy of the system in determining the position of the tennis ball in complex scenes is greatly improved. In actual applications, during the game, whenever the central control machine detects the occurrence of the key event of the ball falling to the ground, the central control machine will record the time point, and extract the position information of the tennis ball in the moving state in the dozens of frames or more frames before and after the time point in the locally stored video stream, that is, the outline of the tennis ball, and finally depict it in the real scene map of the venue, forming a complete real-scene dynamic video and / or real-scene static image, to assist the referee in determining the exact position of the tennis ball at the moment of landing and make a penalty.

[0066] The real-scene assisted penalty judgment method provided by the embodiment of the present invention captures video images and extracts the movement trajectory of the tennis ball through an ordinary camera device, accurately depicts the position information of the tennis ball before and after it lands on the ground into the real-scene map of the venue, and finally generates a real-scene dynamic video and / or a real-scene static image containing the movement trajectory of the tennis ball, effectively improving the accuracy and authenticity of motion analysis, and can assist in determining the exact position of the tennis ball at the moment of landing and make a penalty judgment, thereby solving the technical problems that the existing Hawkeye system including high-speed cameras is expensive and difficult to be popularized in the field of national fitness, and the existing use of ordinary cameras to assist in penalty judgment cannot accurately determine the landing position of the ball due to insufficient frame rate, and cannot achieve the accuracy of a professional Hawkeye system.

[0067] The real-scene assisted penalty judgment method provided by the embodiment of the present invention also includes the step of pre-generating a real-scene map of the venue. Pre-generating the real-scene map of the venue specifically includes: using a Gaussian mixture model to construct a background model of the real scene of the tennis court venue; performing pixel matching between a preset scene scene image and the background model, detecting a foreground image in the scene scene image and removing the foreground image from the tennis court scene image, and obtaining a scene scene map consistent with the real scene of the tennis court venue, such as Figure 2 shown.

[0068] Specifically, this embodiment is based on a Gaussian mixture model (GMM), uses multiple Gaussian distributions to model the color value of each pixel of the actual scene of the tennis court, and compares each pixel value in each frame of the image with the Gaussian component. Through analysis over a period of time (a sequence of video frames), moving objects (including people, balls, etc.) are detected and eliminated, and finally a clean background image that is completely consistent with the actual venue is retained for subsequent frame difference calculations and for drawing the outline of the tennis ball detected later on this actual scene image.

[0069] In one embodiment of the present invention, extracting the tennis ball contours of the moving tennis ball from the multiple target image frames in step S4 specifically includes the following steps which are not shown in the drawings:

[0070] S41, extracting an initial mask image corresponding to each target image frame, which only describes the pixels of the moving objects in the current image.

[0071] S42, performing edge enhancement processing on the moving object in each initial mask image through morphological operation to obtain a standard mask image with a smooth contour of the moving object corresponding to each target image frame.

[0072] S43, performing tennis ball contour detection on the standard mask image of each target image frame to obtain a target mask image corresponding to each target image frame that only contains the tennis ball contour.

[0073] Furthermore, in step S41, an initial mask image corresponding to each target image frame and only describing the pixels of moving objects in the current image is extracted, which specifically includes: performing differential operations on each target image frame and the real scene map of the site to obtain a global difference image corresponding to each target image frame; performing differential operations on each target image frame and a previous image frame adjacent to the current target image frame to obtain a local difference image corresponding to each target image frame; integrating the global difference image and the local difference image corresponding to each target image frame to obtain an initial mask image corresponding to the current target image frame and only describing the pixels of moving objects.

[0074] Specifically, the image is represented in the computer as a multidimensional array, where each element corresponds to the color value (RGB) of a pixel. By calculating the absolute difference between each pixel, each pixel value of the final output image is the absolute difference between the pixel values ​​at the corresponding positions of the two input images, thereby obtaining an initial mask image that only describes the pixels of the moving object.

[0075] The specific formula is: output[i,j]=|src1[i,j]-src2[i,j]|, where i and j are the pixel values ​​at the corresponding positions of the two input images.

[0076] In this embodiment, in order to achieve the best dynamic recognition effect in the entire environment cycle, the present invention optimizes the frame difference implementation method: that is, the frame difference method is used twice in succession, where the current target image frame is subjected to a frame difference with the background frame (i.e., the real scene image of the site) to obtain a global difference, and then the current target image frame is subjected to a frame difference with the adjacent previous image frame to obtain a local difference, and the two differences are integrated to obtain an initial mask image corresponding to the current target image frame that only describes the pixels of the moving object, such as Figure 3 As shown, this can enhance the robustness of detection and reduce the impact of noise.

[0077] Furthermore, in step S42, edge enhancement processing is performed on the moving objects in each initial mask image through morphological operations, including: expanding the moving objects in each initial mask image outward through a first dilation operation to enhance the visibility and edges of the moving objects, corroding the moving objects through an opening operation to remove noise in the image, and then expanding the moving objects outward through a second dilation operation to restore the size of the moving objects.

[0078] Specifically, the initial mask image obtained in step S14 cannot be directly output as a result, because there are still a lot of noises between pixels and objects are hollow or disconnected, and further processing is required. This embodiment performs edge enhancement processing on the moving objects in each initial mask image through the following operations, specifically including:

[0079] c1. The dilation operation will expand the moving object outward. Specifically, the structural element (kernel) is used to cover each pixel area in the image, enhancing the visibility of the object and the edge of the object, especially when there are small gaps or small noises in the image that need to be filled. At the same time, it helps to connect the separated parts of the image, such as helping to connect disconnected objects.

[0080] c2. The opening operation also uses the structural element (kernel) to corrode the object first, remove the small noise in the image, and then restore the size of the object by dilation. This can effectively clean up the outline of the object and make it smoother, and obtain a standard mask image with a smooth outline of the moving object corresponding to each target image frame, such as Figure 4 shown.

[0081] In this process, the present invention constructs different structural elements (kernels) for morphological operations through continuous parameter experiments and performance optimization. Optionally, the structural element can be 5*5, 3*3 (unit: pixel), etc.

[0082] Furthermore, in step S43, performing tennis ball contour detection on the standard mask image of each target image frame includes: performing boundary tracking on the moving object in the standard mask image of each target image frame to find the closed area corresponding to the moving object, storing the contour corresponding to the closed area and / or the pixel points in the contour, and obtaining the contour information of all moving objects in the image; and identifying the contour information of each moving object according to at least one preset tennis ball feature recognition rule, and detecting the tennis ball contour in the standard mask image of each target image frame. In this embodiment, performing boundary tracking on the moving object in the standard mask image to find the closed area corresponding to the moving object specifically includes: starting from each pixel in the standard mask image, using chain coding to traverse along the boundary of the moving object to find adjacent contour pixels until the closed area corresponding to the moving object is found. Among them, the tennis feature recognition rules include: detecting whether the contour area corresponding to each moving object meets the preset area threshold range based on the contour information of the moving object; and / or, detecting whether the contour shape corresponding to each moving object meets the preset shape parameter requirements based on the contour information of the moving object; and / or, performing vector relationship detection on the contour information of the moving object and the tennis contour detected in at least one adjacent continuous target image frame to determine whether the vector relationship of multiple continuous contours conforms to the motion trajectory.

[0083] Specifically, the initial mask image obtained in the above step S14 is the pixel points of all moving objects in a certain video frame. After morphological operations, they have been divided into specific moving objects, including athletes, tennis rackets, tennis balls, etc. In this step, the contour of the corresponding object will be searched based on the standard mask image. The contour is the boundary of the area with the same color or intensity in the image, and the contour that does not meet the target object conditions is filtered out. In this embodiment, it specifically refers to the contour that does not meet the tennis ball feature recognition rules, so as to further determine which contour corresponds to the moving object is the tennis ball.

[0084] d1. Find the contour: First, scan the standard mask image for boundary tracking. Starting from each pixel in the image, use chain coding to traverse along the boundary of the contour to find adjacent contour pixels. After performing point description and finding the closed area, store these contours and the pixels within the contours. In this way, the contour information of all objects in the image can be obtained.

[0085] d1. Filtering: In order to find the information of tennis balls, we filter these contours through specific tennis ball feature recognition rules. For example, contour area feature recognition rule, if it is too large or too small, it is not a tennis ball); contour shape feature recognition rule, if the aspect ratio is too large, it is not a tennis ball; vector relationship feature recognition rule, vector is used to represent a quantity with size and direction, by checking whether the vector relationship of multiple consecutive contours conforms to the motion trajectory, etc., finally a target mask image containing only the tennis ball contour is obtained, such as Figure 5 shown.

[0086] In an embodiment of the present invention, in step S5, the tennis ball contour extracted from multiple target image frames is drawn on a preset venue real scene map, specifically including: performing a bitwise AND operation on the standard mask image of each target image frame and the venue real scene map respectively, so as to merge the tennis ball contour in each standard mask image into the venue real scene map, and obtain a continuous real scene image sequence including only the tennis ball, and generating a real scene dynamic video including the tennis ball movement trajectory corresponding to the tennis ball landing event according to the real scene image sequence; and / or performing a bitwise AND operation on the standard mask image of each target image frame and the same venue real scene map, so as to merge the tennis ball contour in each standard mask image into a venue real scene map, and obtain a real scene static image including the tennis ball movement trajectory corresponding to the tennis ball landing event.

[0087] Specifically, in one embodiment, the mask image containing only the outline of the tennis ball obtained in step S4 can be merged into the pre-obtained mask image such as Figure 2 In the real scene of the court shown in FIG. 1 , a real scene image with only tennis balls is obtained. The above process is repeated for a period of time and all are drawn to the previously obtained Figure 2 In the real scene of the court shown in the figure, a continuous video sequence frame can be obtained, which constitutes a real-life dynamic video corresponding to the tennis ball landing event including the tennis ball's motion trajectory. The information in this video helps the user to accurately observe the tennis ball landing and judge whether the tennis ball is out of bounds.

[0088] Specifically, in another embodiment, the mask image containing only the tennis ball outline of each target image frame obtained in step S4 can be bitwise ANDed with the same field real scene image to merge the tennis ball outlines in each standard mask image into a field real scene image, and obtain a real scene static image including the tennis ball motion trajectory corresponding to the tennis ball landing event, such as Figure 6 The information in this real-life static image helps the user to accurately observe the tennis ball until it hits the ground and determine whether the tennis ball is out of bounds.

[0089] The real-scene assisted penalty method provided by the embodiment of the present invention uses the tennis motion video captured by the real monitoring camera and uses advanced computer vision technologies such as background subtraction, morphological processing and contour detection to identify and track the motion trajectory of the tennis ball in real time. Different from the traditional simulation scene technology, this method automatically analyzes the data of the real venue, extracts the position information before and after the tennis ball lands, and accurately draws it into the real scene map of the venue, and finally generates a real-scene dynamic video and a real-scene static image containing the tennis ball's motion trajectory, effectively improving the accuracy and authenticity of motion analysis and helping referees make decisions.

[0090] The present invention captures video images and extracts the trajectory of tennis balls through ordinary surveillance cameras, depicts the position of the tennis ball before and after it lands in the real scene of the venue, and can assist the referee in determining the exact position of the tennis ball at the moment of landing and making a penalty. Although ordinary surveillance cameras have insufficient frame rates and may miss the key point where the ball lands, the present invention can provide the referee with sufficient information to determine the positional relationship between the ball and the boundary of the venue when it lands by depicting the complete trajectory of tennis balls, so as to accurately implement real-scene assisted penalty judgment.

[0091] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0092] Another embodiment of the present invention further provides a real-scene auxiliary penalty judgment system, the system includes a cloud processing system, a client device and an on-site device, the on-site device includes a central control machine, a display device, at least one camera device and some network auxiliary facilities, wherein:

[0093] The camera device is implemented by a camera, which can be an ordinary camera or a high-speed camera, and the present invention does not limit this. If the camera includes at least two channels, it can be installed on both sides of the court. If there is only one camera, it can be installed on only one side of the court. The camera device is used to shoot the tennis court and the tennis ball and human body located in the tennis court to collect the scene picture in real time.

[0094] The central control machine is used to store the video stream collected by the camera device. Preferably, the central control machine is used to locally store the video stream collected by the camera device, such as a camera.

[0095] The client device is a terminal device installed with referee tools, such as referee applet or application software. The terminal device can be a mobile phone, notebook, tablet computer, etc. The client device is used to trigger auxiliary penalty events according to user requests and send auxiliary penalty requests to the cloud processing system.

[0096] The cloud processing system is responsible for dispatching on-site equipment. Specifically, it is used to send penalty processing commands to the central control machine when receiving auxiliary penalty requests sent by client devices.

[0097] The central control machine is also used to execute the real-scene assisted penalty judgment method described in any of the above embodiments when receiving the penalty judgment processing command sent by the cloud processing system, for example Figure 1 Steps S1-S6 are shown. The penalty result is sent to the cloud processing system;

[0098] The cloud processing system is also used to push the penalty results to the on-site display device and / or client device for display.

[0099] In a specific embodiment, the specific scheduling process of the real-scene auxiliary penalty system is as follows: when a user initiates an auxiliary penalty through the mini program entrance, the request will be sent to the cloud processing system. The cloud processing system notifies the central control machine to perform penalty processing. After the central control machine completes the processing, the penalty result of the penalty processing is sent to the cloud processing system, and the cloud processing system pushes the penalty result to the large screen on site or the mobile phone mini program for display.

[0100] For the system embodiment, since its actual scene assisted penalty judgment process is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment, and it has the corresponding technical effects.

[0101] The real-scene assisted penalty judgment method and system provided by the embodiment of the present invention captures video images and extracts the movement trajectory of the tennis ball through an ordinary camera device, accurately depicts the position information of the tennis ball before and after it lands on the ground into the real-scene map of the venue, and finally generates a real-scene dynamic video and / or a real-scene static image containing the movement trajectory of the tennis ball, effectively improving the accuracy and authenticity of motion analysis, and can assist in determining the exact position of the tennis ball at the moment of landing and make penalties, thereby solving the technical problems that the existing Hawkeye system including high-speed cameras is expensive and difficult to be popularized in the field of national fitness, and that the existing use of ordinary cameras to assist in penalty judgment cannot accurately determine the landing position of the ball due to insufficient frame rate, and cannot achieve the accuracy of a professional Hawkeye system.

[0102] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-scene assisted penalty judgment method, characterized in that: The method comprises: Acquire video data collected by a camera device in real time on a tennis court and tennis balls and human bodies located in the tennis court; Monitoring a tennis ball falling event in the video data, and recording a time point at which the tennis ball falling event occurs in the video data; Acquire multiple target image frames in the video data that are within a preset time length before and after the occurrence time point; extracting tennis ball contours of the moving tennis ball from the plurality of target image frames respectively; Drawing the tennis ball outline extracted from the plurality of target image frames on a preset field real scene image to form a real-scene dynamic video and / or a real-scene static image including a tennis ball movement trajectory corresponding to the tennis ball landing event; An auxiliary penalty is performed based on the real-scene dynamic video and / or the real-scene static image and a penalty result is generated, where the penalty result includes a determination result of whether the tennis ball is out of bounds and the real-scene dynamic video and / or the real-scene static image.

2. The method according to claim 1, characterized in that The method further comprises: pre-generating a real scene map of the site; Pre-generate site real-life pictures, including: Use Gaussian mixture model to construct the background model of the real scene of the tennis court; The preset scene scene image is pixel-matched with the background model, the foreground image in the scene scene image is detected and the foreground image is removed from the tennis court scene image, so as to obtain a field scene image consistent with the actual scene of the tennis court.

3. The method according to claim 1, characterized in that The step of extracting the tennis ball contours of the moving tennis ball from the plurality of target image frames comprises: Extracting the initial mask image corresponding to each target image frame, which only describes the pixels of the moving objects in the current image; The moving objects in each initial mask image are enhanced by morphological operations to obtain a standard mask image with smooth contours of the moving objects corresponding to each target image frame. The tennis ball contour is detected on the standard mask image of each target image frame to obtain a target mask image containing only the tennis ball contour corresponding to each target image frame.

4. The method according to claim 3, characterized in that The extracting of an initial mask image corresponding to each target image frame and describing only pixels of moving objects in the current image includes: Performing a differential operation on each target image frame and the site real scene image to obtain a global difference image corresponding to each target image frame; Performing a differential operation on each target image frame and a previous image frame adjacent to the current target image frame to obtain a local difference image corresponding to each target image frame; The global difference image and the local difference image corresponding to each target image frame are integrated to obtain an initial mask image corresponding to the current target image frame that only describes the pixels of the moving object.

5. The method according to claim 3, characterized in that: The edge enhancement processing of the moving object in each initial mask image by morphological operation includes: The moving objects in each initial mask image are expanded outward through the first dilation operation to enhance the visibility and edges of the moving objects, the moving objects are eroded through the opening operation to remove noise in the image, and the moving objects are expanded outward through the second dilation operation to restore the size of the moving objects.

6. The method according to claim 3, characterized in that: The detecting of tennis ball contour on the standard mask image of each target image frame comprises: Track the boundaries of the moving objects in the standard mask image of each target image frame to find the closed area corresponding to the moving objects, store the contours corresponding to the closed area and / or the pixels within the contours, and obtain the contour information of all the moving objects in the image; The contour information of each moving object is identified according to at least one preset tennis feature recognition rule, and the tennis contour in the standard mask image of each target image frame is detected.

7. The method according to claim 6, characterized in that The tennis feature recognition rules include: Detecting whether the contour area corresponding to each moving object meets a preset area threshold range according to the contour information of the moving object; and / or Detecting whether the contour shape corresponding to each moving object meets the preset shape parameter requirements according to the contour information of the moving object; and / or A vector relationship detection is performed on the contour information of the moving object and the tennis ball contour detected in at least one adjacent continuous target image frame to determine whether the vector relationship of the continuous multiple contours conforms to the motion trajectory.

8. The method according to claim 6, characterized in that Tracking the boundary of the moving object in the standard mask image to find the closed area corresponding to the moving object includes: Starting from each pixel in the standard mask image, chain encoding is used to traverse along the boundary of the moving object to find adjacent contour pixels until a closed area corresponding to the moving object is found.

9. The method according to claim 6, characterized in that Drawing the tennis ball outline extracted from the plurality of target image frames on a preset field real scene image includes: Performing bitwise AND operations on the standard mask images of each target image frame and the real scene image of the venue, respectively, so as to merge the outline of the tennis ball in each standard mask image into the real scene image of the venue, thereby obtaining a continuous real scene image sequence including only the tennis ball, and generating a real scene dynamic video including the movement trajectory of the tennis ball corresponding to the tennis ball landing event according to the real scene image sequence; and / or The standard mask images of each target image frame are bitwise ANDed with the same field real scene image to merge the tennis ball outlines in each standard mask image into a field real scene image, thereby obtaining a real-scene static image including the tennis ball's motion trajectory corresponding to the tennis ball landing event.

10. A real-scene auxiliary penalty judgment system, characterized in that: The system comprises: Cloud processing system, client device, central control machine, display device and at least one camera device; A camera device is used to shoot the tennis court and the tennis balls and human bodies in the tennis court to collect the scene images in real time; The central control machine is used to store the video stream collected by the camera device; The client device is used to trigger an auxiliary penalty event according to a user request and send the auxiliary penalty request to a cloud processing system; The cloud processing system is used to send a penalty processing command to the central control machine when receiving an auxiliary penalty request sent by the client device; The central control machine is further used to execute the real-scene assisted penalty judgment method according to any one of claims 1 to 9 upon receiving the penalty processing command sent by the cloud processing system and send the penalty result to the cloud processing system; The cloud processing system is also used to push the penalty results to the on-site display device and / or client device for display.

Images