Methods, devices, equipment, media, and products for showcasing golf swings.

Abstract

This invention discloses a method, apparatus, device, medium, and product for displaying a golf shot scene, relating to the field of data processing technology. The method includes: performing point cloud semantic classification on the depth point cloud at a given time unit based on camera images, obtaining a region point cloud of at least one target region corresponding to that time unit; performing geometric fitting on the region point clouds of the ball region, the club region, and the ground region, respectively, to determine the structured parameters of the ball region, the club region, and the ground region corresponding to each time unit; determining the shot time from each time unit; selecting a set number of time units centered on the shot time to form a time window, and rendering and displaying the golf shot scene based on the region point clouds and structured parameters of each time unit within the time window. This invention provides a complete three-dimensional display of a golf shot scene.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to methods, devices, equipment, media, and products for displaying golf swing scenes. Background Technology

[0002] Golf demands a high degree of precision in a player's swing. After completing a shot, players generally need to review their swing to identify weaknesses or correct incorrect power generation habits.

[0003] In existing technologies, the main method is to use handheld devices or fixed camera positions on the field to film the player's hitting process. This is cumbersome to operate and has a single, fixed perspective, making it difficult to fully and clearly show the scene details before and after the hit. Summary of the Invention

[0004] This invention provides a method, device, equipment, medium, and product for displaying golf hitting scenes, so as to achieve a complete three-dimensional display of golf sports scenes.

[0005] In a first aspect, embodiments of the present invention provide a method for displaying a golf shot scene, including:

[0006] Acquire camera images and depth point clouds at at least one unit of time during a golf swing event;

[0007] For each unit of time, based on the camera image at that unit of time, point cloud semantic classification is performed on the depth point cloud at that unit of time to obtain the region point cloud of at least one target region corresponding to that unit of time; the target region includes the human body region, the cue stick region, the ball region, and the ground region;

[0008] Geometric fitting is performed on the point clouds of the sphere region, the point clouds of the club region, and the point clouds of the ground region respectively to determine the structured parameters of the sphere region, the club region, and the ground region corresponding to the unit time.

[0009] The striking time is determined from each of the aforementioned unit times based on the structured parameters of the ball region and the structured parameters of the club region corresponding to two adjacent unit times.

[0010] Centered on the moment of the shot, a set number of unit moments are selected to form a time window, and the golf shot scene is rendered and displayed based on the regional point cloud and structured parameters of each unit moment in the time window.

[0011] Secondly, embodiments of the present invention also provide a display device for a golf shot scene, comprising:

[0012] The acquisition module is used to acquire camera images and depth point clouds at at least one unit moment during a golf swing event;

[0013] The classification module is used to perform point cloud semantic classification on the depth point cloud at each unit time based on the camera image at that unit time, to obtain the region point cloud of at least one target region corresponding to that unit time; the target region includes the human body region, the cue stick region, the ball region, and the ground region;

[0014] The fitting module is used to perform geometric fitting on the point cloud of the sphere region, the point cloud of the club region, and the point cloud of the ground region, respectively, to determine the structured parameters of the sphere region, the club region, and the ground region corresponding to the unit time.

[0015] The timing determination module is used to determine the striking time from each of the unit times based on the structured parameters of the ball region and the structured parameters of the club region corresponding to two adjacent unit times.

[0016] The display module is used to select a set number of unit moments centered on the moment of the shot to form a time window, and to render and display the golf shot scene based on the regional point cloud and structured parameters of each unit moment in the time window.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, comprising:

[0018] At least one processor; and

[0019] A memory that is communicatively connected to at least one processor; wherein

[0020] The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method for displaying a golf shot scene provided in any embodiment of the present invention.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute a method for displaying a golf hitting scene according to any embodiment of the present invention.

[0022] Fifthly, embodiments of the present invention also provide a computer program product, characterized in that the computer program product includes a computer program, which, when executed by a processor, implements the method for displaying a golf hitting scene according to any embodiment of the present invention.

[0023] This invention performs semantic classification on depth point clouds based on camera images to distinguish between human body regions, club regions, ball regions, and ground regions within the depth point clouds. It then performs geometric fitting on the ball and club regions to output structured parameters, and determines the moment of impact based on these parameters. A time window is constructed centered on the moment of impact, and a dynamic 3D golf scene is displayed based on the regional point clouds and structured parameters at each moment within the time window, showing how the scene changes over time. Compared to existing technologies that rely on capturing the impact process with equipment, this invention achieves a complete 3D display of the golf impact scene, presenting details of the impact from various perspectives, and dynamically changing over time, thus improving the visualization and practicality of the golf impact scene.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a method for displaying a golf shot scene according to Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart of a method for displaying a golf shot scene according to Embodiment 2 of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a display device for a golf hitting scene according to Embodiment 3 of the present invention;

[0029] Figure 4 This is a structural diagram of an electronic device that implements a method for displaying a golf shot scene according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] The acquisition, storage, and application of iso-camera images and depth point clouds involved in the technical solutions of this invention comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a method for displaying a golf hitting scene according to Embodiment 1 of the present invention. This embodiment is applicable to the display of a golf hitting scene. The method can be executed by a golf hitting scene display device, which can be implemented in hardware and / or software and specifically configured in an electronic device.

[0035] See Figure 1 The methods for demonstrating a golf shot scene include:

[0036] S101. Acquire camera images and depth point clouds at least one unit time during a golf swing event.

[0037] S102. For each unit of time, based on the camera image at the unit of time, perform point cloud semantic classification on the depth point cloud at the unit of time to obtain the region point cloud of at least one target region corresponding to the unit of time; the target region includes the human body region, the cue stick region, the ball region and the ground region.

[0038] S103. Perform geometric fitting on the point cloud of the sphere region, the point cloud of the club region, and the point cloud of the ground region respectively to determine the structured parameters of the sphere region, the club region, and the ground region corresponding to the unit time.

[0039] S104. Determine the striking time from each of the unit times based on the structured parameters of the ball region and the structured parameters of the cue region corresponding to two adjacent unit times.

[0040] S105. Centered on the moment of the shot, select a set number of unit moments to form a time window, and render and display the golf shot scene based on the regional point cloud and structured parameters of each unit moment in the time window.

[0041] In this embodiment, a golf ball hitting event can be an event that includes the player's complete hitting action. The specific start and end points of the event can be set by technicians according to actual needs or practical experience. For example, the start point of the event can be the player standing at the hitting position, and the end point can be the player leaving the hitting position. Alternatively, the start point can be the player's complete swing preparation action, and the end point can be the golf ball being hit, flying, and landing still.

[0042] The camera image can be an image captured by a camera, such as an RGB image captured by an RGB camera. It should be noted that the RGB camera is merely an example, and this invention does not limit the type of camera or the format of the camera image. The depth point cloud can be a point cloud captured by a depth sensor, such as a point cloud captured by a LiDAR. It should be noted that the LiDAR is merely an example, and this invention does not limit the method of acquiring the depth point cloud. The region point cloud can be a point cloud of a corresponding region. The human body region can be the region of the player's human body; the club region can be the region of a golf club; the ball region can be the region of a golf ball; the ground region is the region of the ground where the player stands; in an optional embodiment, the club region may further include a clubhead sub-region and a shaft sub-region.

[0043] Structured parameters can be parameters obtained by geometric fitting of the target area; specifically, the structured parameters of the sphere area can include, but are not limited to, the center position and radius of the sphere; the structured parameters of the club area can include, but are not limited to, the club shaft axis parameters, clubhead position and clubhead orientation; the structured parameters of the ground area can include, but are not limited to, the plane equation parameters of the ground, such as the ground normal vector and position scalar constant.

[0044] The moment of impact can be the instant the golf ball is struck. A time window can include a set number of consecutive unit moments. The golf shot scene can be a three-dimensional scene including the human body, the ball, the club, and the ground.

[0045] Specifically, camera images and depth point clouds are acquired at least one unit of time during a golf swing event; for each unit of time, based on the camera image at that unit of time, point cloud semantic classification is performed on the depth point cloud at that unit of time to obtain the region point cloud of at least one target region corresponding to that unit of time; the target region includes the human body region, the club region, the ball region, and the ground region.

[0046] Geometric fitting is performed on the point clouds of the sphere region, the point clouds of the club region, and the point clouds of the ground region, respectively, to determine the structured parameters of the sphere region, the club region, and the ground region at the given unit time.

[0047] Optionally, geometric fitting is performed on the point cloud of the spherical region to obtain the structured parameters of the spherical region corresponding to the unit time, including:

[0048] Based on the standard golf ball radius and the regional point cloud of the ball's area, the position of the ball's center is fitted; for example, the position of the ball's center can be determined by the following formula:

[0049] ;

[0050] in, Indicates the position of the center of the ball; A point cloud representing a region of a sphere; Indicates the first One point; Indicates the candidate position; This indicates the radius of a standard golf ball.

[0051] Optionally, the club region may include a shaft sub-region and a clubhead sub-region; geometric fitting is performed on the point cloud of the club region to obtain the structured parameters of the club region corresponding to the unit time, including:

[0052] The club axis is fitted based on points in the shaft region of the club area; for example, the club axis can be determined by the following formula:

[0053] ;

[0054] ;

[0055] in, Indicates the axis of the cue stick; This represents a point on the axis of the cue stick; It is a scalar parameter; The direction vector of the cue stick; A point cloud representing a sub-region of the shaft within the club area; Indicates the axis of the candidate sphere; Indicate candidate points; This represents the direction vector of the candidate club shaft.

[0056] The clubhead position can be estimated based on the local point cloud near the end of the club axis, the image detection box of the clubhead, and the clubhead template.

[0057] Optionally, geometric fitting is performed on the regional point cloud of the ground area to obtain the structured parameters of the ground area corresponding to the unit time, including:

[0058] Based on the regional point cloud of the ground area, a ground plane is fitted; for example, the ground plane can be determined by the following formula:

[0059] ;

[0060] ;

[0061] in, Represents the ground plane; Represents the ground normal vector; Represents a positional scalar constant; Represents the candidate ground normal vector; Represents a scalar constant for the candidate position; A regional point cloud representing a ground area.

[0062] Based on the structured parameters of the ball region and the club region corresponding to two adjacent unit times, the striking time is determined from each unit time. A set number of unit times are selected as the center to form a time window. The set number of unit times can be set independently by technicians based on actual needs or practical experience; this invention does not limit this. For each unit time in the time window, the golf striking scene corresponding to that unit time is rendered based on the regional point cloud and structured parameters of each target region at that unit time. The golf striking scenes at each unit time are encapsulated and displayed according to the time sequence within the time window.

[0063] Optionally, determining the striking time from each unit time based on the structured parameters of the ball region and the structured parameters of the club region corresponding to two adjacent unit times includes: determining the clubhead velocity at the second time based on the clubhead position at the first time and the clubhead position at the second time; the first time being the previous unit time of the second time; calculating the target distance between the clubhead position and the center of the golf ball at the second time based on the clubhead position and the center of the golf ball at the second time; determining whether the second time is a striking time based on the clubhead velocity and the target distance; if the second time is not a striking time, updating the first time to the second time, updating the second time to the next time of the second time, and returning to the step of determining the clubhead velocity at the second time based on the clubhead position at the first time and the clubhead position at the second time, until the second time is a striking time.

[0064] Specifically, the earliest unit time in the time sequence is taken as the first time, and the unit time adjacent to the earliest unit time and located after the earliest unit time is taken as the second time. The clubhead speed at the second time is determined based on the clubhead position at the first time, the clubhead position at the second time, and the time interval between the unit times. For example, the clubhead speed can be determined using the following formula:

[0065] ;

[0066] in, This indicates the clubhead speed at the second moment; Indicates the clubhead position at the second moment; Indicates the clubhead position at the first moment; Indicates the duration interval.

[0067] For example, the target distance can be determined using the following formula:

[0068] ;

[0069] in, Indicates the target distance at the second moment; This indicates the position of the ball's center at the second moment.

[0070] The clubhead velocity and target distance at the second moment are weighted and calculated to obtain the impact moment score. If the impact moment score is greater than a preset score threshold, the second moment is determined to be an impact moment; otherwise, the second moment is determined not to be an impact moment. If the second moment is not an impact moment, the first moment is updated to the second moment, and the second moment is updated to the next moment after the second moment. The process then returns to the step of determining the clubhead velocity at the second moment based on the clubhead position at the first moment and the clubhead position at the second moment, until the second moment is an impact moment.

[0071] Understandably, by adopting the above technical solution, the clubhead velocity is calculated based on the clubhead position at two adjacent unit moments; the distance between the clubhead and the center of the golf ball is calculated based on the clubhead position and the center of the golf ball at the same moment; and the moment of impact is determined moment by moment based on the clubhead velocity and the distance between the clubhead and the center of the ball, which can improve the accuracy of determining the moment of impact.

[0072] Optionally, determining whether the second moment is a shot moment based on the clubhead speed and target distance at the second moment includes: acquiring the impact sound intensity at the second moment; determining the shot moment score at the second moment based on the clubhead speed, target distance, and impact sound intensity at the second moment; and determining the second moment as a shot moment if the shot moment score at the second moment is greater than a set threshold.

[0073] The impact sound intensity can be the sound intensity produced by the impact action. In a specific real-time method, sound acquisition equipment can be deployed on the golf course to collect audio at each unit moment of the golf impact event. Based on the pre-measured range of the impact sound audio frequency band, the audio is filtered to obtain the impact audio, and the intensity of the impact audio is calculated to obtain the impact sound intensity. The clubhead speed, target distance, and impact sound intensity at the second moment are weighted and calculated to obtain the impact moment score at the second moment. For example, the impact moment score can be determined by the following formula:

[0074] ;

[0075] in, The score indicates the moment of impact at the second moment; Indicates the weight of clubhead speed; Distance weight; To prevent constant terms with a denominator of zero; Sound intensity weighting; This indicates the sound intensity of the ball being struck at the second moment.

[0076] Understandably, by adopting the above technical solution, the intensity of the impact sound is obtained, and combined with the sound intensity, clubhead speed, and target distance, the impact moment score is comprehensively determined. This avoids misjudging the impact moment due to abnormalities such as obstruction and blurring caused by the high-speed movement of the club, thereby improving the accuracy of determining the impact moment.

[0077] This invention performs semantic classification on depth point clouds based on camera images to distinguish between human body regions, club regions, ball regions, and ground regions within the depth point clouds. It then performs geometric fitting on the ball and club regions to output structured parameters, and determines the moment of impact based on these parameters. A time window is constructed centered on the moment of impact, and a dynamic 3D golf scene is displayed based on the regional point clouds and structured parameters at each moment within the time window, showing how the scene changes over time. Compared to existing technologies that rely on capturing the impact process with equipment, this invention achieves a complete 3D display of the golf impact scene, presenting details of the impact from various perspectives, and dynamically changing over time, thus improving the visualization and practicality of the golf impact scene.

[0078] Example 2

[0079] Figure 2 This is a flowchart of a method for displaying a golf hitting scene according to Embodiment 2 of the present invention. The present invention has made additional optimizations based on the technical solutions of the above embodiments.

[0080] Furthermore, before "showing the golf shot scene", the following is added: "Using a trained depth completion model, the region point cloud of each target region in the time window is depth-completed, and the region point cloud of each target region is updated to the depth-completed point cloud."

[0081] It should be noted that for any parts not described in detail in the embodiments of the present invention, please refer to the description in the foregoing embodiments.

[0082] See Figure 2 The methods for demonstrating a golf shot scene include:

[0083] S201. Acquire camera images and depth point clouds at least one unit time during a golf swing event.

[0084] S202. For each unit of time, based on the camera image at the unit of time, perform point cloud semantic classification on the depth point cloud at the unit of time to obtain the region point cloud of at least one target region corresponding to the unit of time; the target region includes the human body region, the cue stick region, the ball region and the ground region.

[0085] S203. Perform geometric fitting on the point cloud of the sphere region, the point cloud of the club region, and the point cloud of the ground region respectively, and determine the structured parameters of the sphere region, the club region, and the ground region corresponding to the unit time.

[0086] S204. Determine the striking time from each of the unit times based on the structured parameters of the ball region and the structured parameters of the club region corresponding to two adjacent unit times.

[0087] S205. Using the moment of impact as the center, select a set number of unit moments to form a time window.

[0088] S206. Using the trained depth completion model, perform depth completion on the point clouds of each target region in the time window, and update the point clouds of each target region to the point clouds after depth completion.

[0089] S207. Render and display the golf shot scene based on the regional point cloud and structured parameters at each unit time in the time window.

[0090] In this embodiment, the depth completion model can be used to complete the point cloud of each target region to fill in the occluded areas; for example, it can complete the point cloud of the ball region occluded by the golf club, or the point cloud of the golf club region occluded by the human body, etc.

[0091] Optionally, the deep completion model is trained as follows: Using an initial completion model, the point clouds of each target region in the sample hitting event are completed based on the point clouds of each target region, sample camera images, structured parameters of the cue region, and structured parameters of the ball region, resulting in completed point clouds for each target region. Based on the completed point clouds and structured parameters of the ball region at each unit time in the sample hitting event, the sphericity loss of the ball is calculated. Based on the point clouds and structured parameters of the cue region at each unit time in the sample hitting event, the cue axis loss is calculated. Based on the completed point clouds of each target region at every two adjacent unit time points, the temporal continuity loss is calculated. The true loss between the completed point clouds of each target region and the true complete point cloud is calculated. The sphericity loss of the ball, the cue axis loss, the temporal continuity loss, and the true loss are fused to obtain the target loss. The initial completion model is trained with the goal of minimizing the target loss value to obtain the deep completion model.

[0092] The initial completion model can be a model whose parameters have not been trained. The spherical loss characterizes the difference between the shape of the completed point cloud in the spherical region and the shape of a sphere; the club axis loss characterizes the distance between the completed point cloud in the club region and the club axis; the temporal continuity loss characterizes the difference between the completed point clouds of the same target region at adjacent time points; and the truth loss characterizes the difference between the completed point cloud and the true complete point cloud. The true complete point cloud can be the true complete point cloud in the sample shot event.

[0093] Specifically, the point cloud of the target region, the sample camera image, the structured parameters of the club region, and the structured parameters of the ball region at each unit time of the sample hitting event are input into the initial completion model to obtain the completed point cloud of the target region at each unit time of the sample hitting event output by the initial completion model; based on the completed point cloud and structured parameters of the ball region at each unit time of the sample hitting event, the sphericity loss of the ball is calculated; for example, the sphericity loss of the ball can be determined by the following formula:

[0094] ;

[0095] in, This indicates the loss of spherical shape. express The first point cloud of the completed spherical region at time One point.

[0096] For example, the club shaft loss can be determined using the following formula:

[0097] ;

[0098] in, Indicates loss of the cue shaft; express The first point cloud in the moment cue area One point.

[0099] For example, the temporal continuity loss can be determined using the following formula:

[0100] ;

[0101] in, This represents the loss of temporal continuity. express The first complete point cloud at all times One point; express The first complete point cloud at all times One point.

[0102] Calculate the distance difference between each point in the completed point cloud of each target region and the corresponding point in the real complete point cloud, and accumulate the distance differences of each point to obtain the true loss;

[0103] For example, the target loss can be determined using the following formula:

[0104] ;

[0105] in, Loss to the target; This is a real loss; The weight representing the loss due to the sphericity of the sphere; The weight representing the loss along the cue shaft; The weights represent the loss of temporal continuity.

[0106] The initial completion model is trained with the goal of minimizing the target loss value to obtain a deep completion model.

[0107] In an optional embodiment, the rigid body transformation loss of the clubhead can also be calculated for the completed point cloud of the clubhead sub-region within the club region; for example, the rigid body transformation loss of the clubhead can be determined by the following formula:

[0108] ;

[0109] Among them, the This represents the loss due to rigid body transformation at the rod head; Indicate The first point cloud in the completed point cloud of the pole head sub-region at a given moment One point; express The first point cloud in the completed point cloud of the pole head sub-region at a given moment One point; express The orthogonal rotation matrix at time step; express The translation vector at time step.

[0110] Based on the weights of the clubhead rigid body transformation loss, the weighted result of the clubhead rigid body transformation loss is fused into the target loss to obtain the latest target loss. The initial completion model is then trained with the goal of minimizing the value of the latest target loss to obtain the deep completion model.

[0111] Understandably, the above technical solution constrains the sphere to maintain a standard spherical shape through spherical shape loss; ensures the club's slender structure through club axis loss; guarantees motion continuity through temporal loss; and ensures realistic loss closely matches actual data. This multi-constraint collaborative optimization model significantly reduces completion distortion, improves completion accuracy and rationality, and ensures that the completed data conforms to the laws of golf, providing reliable data support for high-quality 3D visualization.

[0112] Optionally, the step of performing point cloud semantic classification on the depth point cloud at the unit time based on the camera image at the unit time to obtain the region point cloud of at least one target region corresponding to the unit time includes: spatially aligning the depth point cloud at the unit time with the camera image at the unit time to obtain the corresponding position of each point in the depth point cloud in the camera image; performing semantic segmentation on the camera image to obtain at least one target region; and for each target region, determining the point cloud formed by the points whose corresponding positions are located in the target region as the region point cloud of the target region.

[0113] Specifically, for each point in the depth point cloud, the point is transformed into the camera coordinate system using the extrinsic parameter matrix between the depth sensor and the image sensor, resulting in the first transformed point of the point in the camera coordinate system. For example, the first transformed point can be determined using the following formula:

[0114] ;

[0115] in, Indicates the first transition point; Represents the extrinsic parameter matrix; This represents a point in the depth point cloud in the depth coordinate system.

[0116] The first transformation point is transformed into the image coordinate system using the camera intrinsic parameter matrix to obtain the second transformation point; for example, the position of the second transformation point can be determined by the following formula:

[0117] ;

[0118] ;

[0119] in, Indicates the x-coordinate of the second transformation point; Indicates the ordinate of the second transformation point; Indicates the camera's horizontal focal length; Indicates the camera's longitudinal focal length; Indicates the x-coordinate of the principal point; Indicates the ordinate of the principal point; Represents the x-coordinate of the first transformation point; This represents the ordinate of the first transformation point; This represents the depth coordinates of the first transformation point.

[0120] The position of the second conversion point is determined as the corresponding position of the point in the camera image. The camera image is semantically segmented using a semantic segmentation model to obtain at least one target region. For each target region, the point cloud formed by the points whose corresponding positions are located within the target region is determined as the region point cloud of the target region.

[0121] It is understandable that by adopting the above technical solution, spatial alignment of the depth point cloud with the camera image is achieved to realize a one-to-one correspondence between 3D points and image pixels; by semantically segmenting the camera image to obtain each target region, and then determining the regional point cloud of each target region through the correspondence between 3D points and image pixels, the accuracy of determining the regional point cloud can be improved.

[0122] This invention uses a trained depth completion model to perform depth completion on the point clouds of each target region, thereby filling in the occluded or missing parts of the point clouds of each target region, restoring the complete three-dimensional contour of each target region, improving the integrity of the point clouds, and thus enhancing the three-dimensional display integrity of the golf hitting scene.

[0123] Example 3

[0124] Figure 3 This is a schematic diagram of a golf hitting scene display device provided in Embodiment 3 of the present invention. This embodiment of the invention is applicable to the display of golf hitting scenes. The device can execute a golf hitting scene display method and can be implemented in hardware and / or software. The device can be configured in an electronic device.

[0125] See Figure 3 The display device for a golf shot scene shown includes an acquisition module 301, a classification module 302, a fitting module 303, a time determination module 304, and a display module 305, wherein...

[0126] The acquisition module 301 is used to acquire camera images and depth point clouds at least one unit moment in a golf swing event;

[0127] The classification module 302 is used to perform point cloud semantic classification on the depth point cloud at each unit time based on the camera image at the unit time, so as to obtain the region point cloud of at least one target region corresponding to the unit time; the target region includes human body region, club region, ball region and ground region;

[0128] The fitting module 303 is used to perform geometric fitting on the point cloud of the sphere region, the point cloud of the club region, and the point cloud of the ground region, respectively, to determine the structured parameters of the sphere region, the club region, and the ground region corresponding to the unit time.

[0129] The timing determination module 304 is used to determine the striking time from each of the unit times based on the structured parameters of the ball region and the structured parameters of the club region corresponding to two adjacent unit times respectively;

[0130] The display module 305 is used to select a set number of unit moments centered on the moment of the shot to form a time window, and to render and display the golf shot scene based on the regional point cloud and structured parameters of each unit moment in the time window.

[0131] This invention performs semantic classification on depth point clouds based on camera images to distinguish between human body regions, club regions, ball regions, and ground regions within the depth point clouds. It then performs geometric fitting on the ball and club regions to output structured parameters, and determines the moment of impact based on these parameters. A time window is constructed centered on the moment of impact, and a dynamic 3D golf scene is displayed based on the regional point clouds and structured parameters at each moment within the time window, showing how the scene changes over time. Compared to existing technologies that rely on capturing the impact process with equipment, this invention achieves a complete 3D display of the golf impact scene, presenting details of the impact from various perspectives, and dynamically changing over time, thus improving the visualization and practicality of the golf impact scene.

[0132] Optionally, the time determination module 304 includes:

[0133] The velocity determination unit is used to determine the clubhead velocity at the second moment based on the clubhead position at the first moment and the clubhead position at the second moment; the first moment is the previous unit moment of the second moment;

[0134] The position determination unit is used to calculate the target distance between the clubhead position and the golf ball center position at the second moment based on the clubhead position and the golf ball center position at the second moment.

[0135] The timing determination unit is used to determine whether the second moment is the moment of impact based on the clubhead speed and target distance at the second moment.

[0136] The loop unit is configured to update the first time to the second time if the second time is not the time of impact, update the second time to the next time after the second time, and return to the step of determining the clubhead velocity at the second time based on the clubhead position at the first time and the clubhead position at the second time, until the second time is the time of impact.

[0137] Optionally, the time determination unit is specifically used for:

[0138] Obtain the impact sound intensity at the second moment;

[0139] The score for the moment of impact at the second moment is determined based on the clubhead speed, target distance, and impact sound intensity at the second moment.

[0140] If the score of the second moment of impact is greater than the set threshold, then the second moment is determined as the moment of impact.

[0141] Optionally, the device further includes:

[0142] The depth completion module is used to perform depth completion on the point clouds of each target region in the time window using a trained depth completion model, and update the point clouds of each target region with the depth-completed point clouds.

[0143] Optionally, the device further includes:

[0144] The initial completion module is used to complete the regional point cloud of each target region in the sample ball-hitting event based on the regional point cloud of each target region in the sample ball-hitting event, the sample camera image, the structured parameters of the club region and the structured parameters of the ball region, and obtain the completed point cloud of each target region.

[0145] The first loss calculation module is used to calculate the spherical loss of the sphere based on the completed point cloud and structured parameters of the sphere region at each unit time in the sample ball-hitting event.

[0146] The second loss calculation module is used to calculate the club axis loss based on the regional point cloud and structured parameters of the club area at each unit time in the sample ball-hitting event.

[0147] The third loss calculation module is used to calculate the temporal continuity loss based on the completed point cloud of each target region at every two adjacent unit time points.

[0148] The fourth loss calculation module is used to calculate the actual loss between the completed point cloud and the real complete point cloud of each target region.

[0149] The loss fusion module is used to fuse the spherical loss of the sphere, the axis loss of the cue stick, the temporal continuity loss, and the true loss to obtain the target loss;

[0150] The training module is used to train the initial completion model with the goal of minimizing the target loss value, so as to obtain a deep completion model.

[0151] Optional, the classification module 302 is specifically used for:

[0152] Spatially align the depth point cloud at the unit time with the camera image at the unit time to obtain the corresponding position of each point in the depth point cloud in the camera image.

[0153] The camera image is semantically segmented to obtain at least one target region;

[0154] For each target region, the point cloud formed by the points located at the corresponding positions within the target region is determined as the regional point cloud of the target region.

[0155] The golf hitting scene display device provided in this embodiment of the invention can execute the golf hitting scene display method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the golf hitting scene display method.

[0156] Example 4

[0157] Figure 4A schematic diagram of a display device 410 for implementing embodiments of the present invention, depicting a golf swing scene, is shown. The display device for the golf swing scene is intended to represent various forms of digital computers, such as laptops, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The display device for the golf swing scene can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0158] like Figure 4 As shown, the golf swing scene display device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 and a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 can also store various programs and data required for the operation of the golf swing scene display device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0159] Multiple components in the golf swing scene display device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, optical disk, etc.; and a communication unit 419, such as a network card, modem, wireless transceiver, etc. The communication unit 419 allows the golf swing scene display device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0160] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as the method of demonstrating a golf swing scene.

[0161] In some embodiments, the method for displaying a golf swing scene can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or installed on the golf swing scene display device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the golf swing scene display method described above can be performed. Alternatively, in other embodiments, processor 411 can be configured to perform the golf swing scene display method by any other suitable means (e.g., by means of firmware).

[0162] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable golf swing scene display device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0164] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0165] To provide user interaction, the systems and techniques described herein can be implemented on a golf swing scene display device, which includes: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the golf swing scene display device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0166] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0167] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0168] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various corrections, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for displaying a golf shot scene, characterized in that, The method includes: Acquire camera images and depth point clouds at at least one unit of time during a golf swing event; For each unit of time, based on the camera image at that unit of time, point cloud semantic classification is performed on the depth point cloud at that unit of time to obtain the region point cloud of at least one target region corresponding to that unit of time; the target region includes the human body region, the cue stick region, the ball region, and the ground region; Geometric fitting is performed on the point clouds of the sphere region, the point clouds of the club region, and the point clouds of the ground region respectively to determine the structured parameters of the sphere region, the club region, and the ground region corresponding to the unit time. The striking time is determined from each of the aforementioned unit times based on the structured parameters of the ball region and the structured parameters of the club region corresponding to two adjacent unit times. Centered on the moment of the shot, a set number of unit moments are selected to form a time window, and the golf shot scene is rendered and displayed based on the regional point cloud and structured parameters of each unit moment in the time window.

2. The method according to claim 1, characterized in that, The step of determining the striking time from each unit time based on the structured parameters of the ball region and the structured parameters of the club region corresponding to two adjacent unit times includes: The clubhead velocity at the second moment is determined based on the clubhead position at the first moment and the clubhead position at the second moment; the first moment is the previous unit moment of the second moment. Calculate the target distance between the clubhead position and the center of the golf ball at the second moment, based on the clubhead position and the center of the golf ball position at the second moment. Based on the clubhead speed and target distance at the second moment, determine whether the second moment is the moment of impact; If the second moment is not the moment of impact, then the first moment is updated to the second moment, and the second moment is updated to the moment after the second moment. Then, return to the step of determining the clubhead speed at the second moment based on the clubhead position at the first moment and the clubhead position at the second moment, until the second moment is the moment of impact.

3. The method according to claim 2, characterized in that, The step of determining whether the second moment is the moment of impact based on the clubhead speed and target distance at the second moment includes: Obtain the impact sound intensity at the second moment; The score for the moment of impact at the second moment is determined based on the clubhead speed, target distance, and impact sound intensity at the second moment. If the score of the second moment of impact is greater than the set threshold, then the second moment is determined as the moment of impact.

4. The method according to claim 1, characterized in that, Before showcasing a golf shot, it also includes: The trained deep completion model is used to perform deep completion on the point clouds of each target region in the time window, and the point clouds of each target region are updated with the deep-completed point clouds.

5. The method according to claim 4, characterized in that, The depth completion model is trained in the following way: Using an initial completion model, the point clouds of each target region in the sample ball-hitting event are completed based on the regional point clouds of each target region, the sample camera image, the structured parameters of the club region, and the structured parameters of the ball region, to obtain the completed point clouds of each target region. The sphericity loss of the sphere is calculated based on the completed point cloud and structured parameters of the sphere region at each unit time in the sample ball-hitting event. The club axis loss is calculated based on the regional point cloud and structured parameters of the club area at each unit time in the sample ball-hitting event. The temporal continuity loss is calculated based on the completed point cloud of each target region at every two adjacent unit time points. Calculate the true loss between the completed point cloud and the true complete point cloud for each target region; The target loss is obtained by fusing the spherical loss of the sphere, the axis loss of the club shaft, the temporal continuity loss, and the actual loss. With the goal of minimizing the target loss, the initial completion model is trained to obtain a deep completion model.

6. The method according to claim 1, characterized in that, The step of performing point cloud semantic classification on the depth point cloud at the unit time based on the camera image at the unit time to obtain the region point cloud of at least one target region corresponding to the unit time includes: Spatially align the depth point cloud at the unit time with the camera image at the unit time to obtain the corresponding position of each point in the depth point cloud in the camera image. The camera image is semantically segmented to obtain at least one target region; For each target region, the point cloud formed by the points located at the corresponding positions within the target region is determined as the regional point cloud of the target region.

7. A display device for a golf swing scene, characterized in that, The device includes: The acquisition module is used to acquire camera images and depth point clouds at at least one unit moment during a golf swing event; The classification module is used to perform point cloud semantic classification on the depth point cloud at each unit time based on the camera image at that unit time, to obtain the region point cloud of at least one target region corresponding to that unit time; the target region includes the human body region, the cue stick region, the ball region, and the ground region; The fitting module is used to perform geometric fitting on the point cloud of the sphere region, the point cloud of the club region, and the point cloud of the ground region, respectively, to determine the structured parameters of the sphere region, the club region, and the ground region corresponding to the unit time. The timing determination module is used to determine the striking time from each of the unit times based on the structured parameters of the ball region and the structured parameters of the club region corresponding to two adjacent unit times. The display module is used to select a set number of unit moments centered on the moment of the shot to form a time window, and to render and display the golf shot scene based on the regional point cloud and structured parameters of each unit moment in the time window.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for displaying a golf shot scene as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for displaying a golf shot scene as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method for displaying a golf shot scene as described in any one of claims 1-6.

Images