Pick ball automatic scoring system based on multi-view high-speed camera shooting and AI identification

Abstract

The invention relates to the technical field of sports event auxiliary decision, and discloses a multi-view high-speed camera shooting and AI identification-based Pick ball automatic scoring system, which comprises a data acquisition subsystem, a data processing subsystem, a system scheduling and fusion decision subsystem and a result output module. The core of the system is a dynamic refocusing mechanism triggered by an acoustic event: the system detects key acoustic events such as ball hitting and the like in real time through a distributed microphone, and accurately calculates the three-dimensional position and timestamp of the events; and the acoustic event information is used as a trigger signal to guide the system to generate a control instruction, so that the visual information processing module is switched from a low-power-consumption conventional monitoring mode to a high-precision fine analysis mode. The system fuses the information of the acoustic mode and the visual mode, and a decision is generated according to a preset competition rule base. According to the invention, the power consumption of the system can be reduced, and the multi-modal information is used for mutual evidence, so that the accuracy and robustness of judgment are improved, and the real-time performance of processing is ensured.

Description

Technical Field

[0001] This invention relates to the field of sports event auxiliary adjudication technology, specifically to an automatic scoring system for pickleball based on multi-view high-speed cameras and AI recognition. Background Technology

[0002] With the deep integration of information technology and competitive sports, automated scoring and adjudication systems are playing an increasingly important role in improving the fairness of competition, enhancing its entertainment value, and providing data-driven analysis. In fast-paced, highly competitive ball sports such as pickleball, accurately capturing and interpreting every key event is a core requirement of automated systems.

[0003] Currently, mainstream technical solutions primarily rely on multiple cameras deployed around the field, using computer vision technology to continuously analyze the video streams to track players and moving balls and identify events. However, such purely vision-based solutions face inherent technical bottlenecks in practical applications. To ensure that no critical moments occurring at high speeds are missed (such as the precise landing point of the ball or whether a violation of the net is committed), the system must perform uninterrupted, high-intensity computational processing on multiple high-frame-rate, high-resolution video streams. This continuous high-intensity processing mode places stringent demands on the performance of computing hardware, significantly increasing system construction costs and resulting in high energy consumption, thus limiting the large-scale application of this technology.

[0004] More importantly, even with significant computational resources, a single source of visual information proves extremely inadequate in the complex and ever-changing environment of a match. Temporary occlusions caused by athletes' bodies or rackets, motion blur from a high-speed ball, and complex variations in lighting conditions can all lead to misjudgments or missed judgments by visual algorithms. For example, the system may struggle to accurately determine the actual point of contact between the ball and the racket, or become confused when multiple ball trajectories intersect, making it difficult to fully guarantee the accuracy and robustness of the rulings.

[0005] This approach of performing in-depth analysis on massive amounts of video data also makes it difficult to achieve both high real-time performance and demanding processing speed. High-precision analysis algorithms often require longer processing times, which directly conflicts with the need for immediate decisions in sports competitions. Sacrificing analytical accuracy for speed would damage the fundamental value of the system; conversely, processing delays would render the system meaningless as a real-time auxiliary tool. Given these factors, existing technologies generally struggle to achieve an ideal balance and synergistic optimization between computational cost-effectiveness, decision accuracy, and real-time processing. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an automatic scoring system for pickleball based on multi-view high-speed photography and AI recognition, which solves the problem of achieving an ideal balance and synergistic optimization among high-precision analysis, real-time processing, and computational cost-effectiveness.

[0007] To achieve the above objectives, the present invention provides an automatic scoring system for pickleball based on multi-view high-speed video recording and AI recognition. This system includes: a data acquisition subsystem, a data processing subsystem, a system scheduling and fusion adjudication subsystem, and a result output and management module.

[0008] The data acquisition subsystem is used to acquire multimodal raw data in real time from the pickleball competition field. The multimodal raw data includes multi-channel synchronized video stream data and multi-channel synchronized audio stream data.

[0009] The data processing subsystem is used to receive and analyze the multimodal raw data to extract structured acoustic event information and visual primitive information.

[0010] The core functions of the system scheduling and fusion adjudication subsystem are as follows: First, based on the acoustic event information output by the data processing subsystem, control commands are generated and sent to the data processing subsystem to dynamically adjust its working mode and resource allocation; second, the acoustic event information and visual primitive information are fused, and logical reasoning is performed based on a preset competition rule base to finally generate the adjudication result.

[0011] The result output and management module is used to receive the adjudication result and convert it into a specified format for output or storage.

[0012] As a preferred technical solution, the data processing subsystem includes an acoustic event processing module and a visual information processing module. The acoustic event processing module detects, classifies, and locates key acoustic events in real time from multi-channel synchronized audio stream data to output the structured acoustic event information. The visual information processing module, under the scheduling of the control commands, analyzes multi-channel synchronized video stream data to extract the visual primitive information.

[0013] Furthermore, the acoustic event processing module includes a signal preprocessing unit, an event detection and classification unit, and a spatiotemporal localization unit. The spatiotemporal localization unit uses cross-correlation or generalized cross-correlation algorithms to calculate estimates of the time delay difference between event signals arriving at different microphones. Subsequently, based on this time delay difference and the pre-calibrated three-dimensional positions of each microphone... Establish a location of the sound source A system of nonlinear equations. If the microphone is taken as the reference, then the first equation in the system... The equations are: ; in, A mathematical expression for calculating the Euclidean distance between two points in three-dimensional space; Specifically refers to the distance between the sound source and the reference microphone; Location of the sound source coordinates For the first One microphone coordinates Let be the speed of sound propagation. The location of the sound source can be obtained by numerically solving this system of equations. The estimated value. The precise timestamp of the event. The result can be obtained by calculating using the following formula: ; in, It is the moment when the reference microphone detects the peak of the signal energy; For reference microphone position; Denotes the Euclidean norm; The straight-line distance between the sound source location and the reference microphone location.

[0014] As another preferred technical solution, the visual information processing module has a dual-modal working mechanism, including a conventional monitoring unit and a refined analysis unit. The conventional monitoring unit is activated in the system default state, operating in a low-resource-consumption manner, downsampling the video stream and performing lightweight object detection. The refined analysis unit is activated upon receiving control commands from the system scheduling and fusion adjudication subsystem, extracting a full-resolution, high-frame-rate original image frame sequence within a specified spatiotemporal range from the circular buffer, and performing computationally intensive tasks, such as multi-view... Figure 3 Human pose estimation and precise sphere segmentation and 3D reconstruction are used to extract high-fidelity visual primitive information.

[0015] Furthermore, the system scheduling and fusion adjudication subsystem includes a dynamic refocusing control module and a multimodal information fusion module. The dynamic refocusing control module contains an event filtering unit that initiates control command generation only when the received acoustic event category belongs to a preset set of key event categories. The generated control command is a two-part data structure: one part uses the acoustic event occurrence timestamp... A time window centered on the location of the acoustic event source, and a time window centered on the location of the acoustic event source. A three-dimensional spatial region of interest centered on a preset size.

[0016] As another preferred technical solution, the multimodal information fusion module includes a causal event graph construction unit and a rule reasoning and adjudication unit. The causal event graph construction unit organizes discrete acoustic event information and visual primitive information into a dynamically updated directed acyclic graph with spatiotemporal and logical relationships. The graph's node set includes acoustic event nodes and visual primitive nodes, while the edge set includes temporal edges connecting continuous states of the same physical object, and associative edges connecting spatiotemporally strongly correlated heterogeneous nodes. The rule reasoning and adjudication unit performs logical queries and traversals on this graph to determine competition events.

[0017] This invention provides an automatic scoring system for pickleball based on multi-view high-speed video recording and AI recognition. It offers the following advantages: 1. This invention utilizes a dynamic refocusing mechanism triggered by acoustic events, enabling the system to perform lightweight routine monitoring under normal conditions. Only when a critical acoustic event is detected does the system selectively activate high-precision visual analysis, processing high-fidelity data stored in a circular buffer within a defined spatiotemporal range. This on-demand processing mode significantly reduces the load of continuous computation on massive amounts of video data, thereby optimizing computing hardware configuration requirements and reducing the overall system energy consumption.

[0018] 2. This invention integrates spatiotemporal positioning information provided by the acoustic event processing module with three-dimensional visual primitive information extracted by the visual information processing module for adjudication. Acoustic information provides a precise temporal and spatial reference for visual analysis, while multi-view visual information supports accurate three-dimensional reconstruction of key actions. The data from the two modalities corroborate each other, effectively overcoming misjudgments caused by single visual solutions under conditions of target occlusion, high-speed motion blur, or complex lighting, thereby improving the reliability of the adjudication results.

[0019] 3. This invention employs an acoustic-first, visual-focused technical strategy. Acoustic event detection is extremely fast, enabling immediate triggering of subsequent processing upon event occurrence. Instead of continuously analyzing all video frames, the system focuses computational resources on brief, critical spatiotemporal segments determined by acoustic events, thereby achieving rapid, high-precision analysis and logical reasoning, ensuring real-time response and adjudication capabilities for competition events. Attached Figure Description

[0020] Figure 1 This is a system framework diagram of the present invention; Figure 2 This is a framework diagram of the acoustic event processing module of the present invention; Figure 3 This is a framework diagram of the visual information processing module of the present invention; Figure 4 This is a framework diagram of the multimodal information fusion module of the present invention; Figure 5This is a schematic diagram of the dynamic refocusing mechanism of the present invention.

[0021] The system comprises: 10. Data acquisition subsystem; 20. Data processing subsystem; 30. System scheduling and fusion adjudication subsystem; and 40. Result output and management module. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please refer to the appendix. Figure 1 This invention provides an automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition. This system is used to automatically identify and adjudicate events during pickleball matches.

[0024] The automatic scoring system for pickleball, based on multi-view high-speed cameras and AI recognition, includes: a data acquisition subsystem 10, a data processing subsystem 20, a system scheduling and fusion adjudication subsystem 30, and a result output and management module 40. The data acquisition subsystem 10, data processing subsystem 20, system scheduling and fusion adjudication subsystem 30, and result output and management module 40 are interconnected.

[0025] Specifically, the data acquisition subsystem 10 is used to acquire multimodal raw data from the pickleball court in real time. The multimodal raw data includes multi-channel synchronized video stream data and multi-channel synchronized audio stream data.

[0026] The input terminal of the data processing subsystem 20 is connected to the output terminal of the data acquisition subsystem 10. The data processing subsystem 20 is used to receive multimodal raw data and analyze and process the data to extract structured acoustic event information and visual primitive information.

[0027] The information input terminal of the system scheduling and fusion adjudication subsystem 30 is connected to the information output terminal of the data processing subsystem 20, and its control output terminal is connected to the control input terminal of the data processing subsystem 20. The system scheduling and fusion adjudication subsystem 30 generates control commands based on the acoustic event information output by the data processing subsystem 20 and sends them to the data processing subsystem 20 to dynamically adjust its operating mode and resource allocation. The system scheduling and fusion adjudication subsystem 30 also fuses acoustic event information with visual primitive information, performs logical reasoning based on a preset competition rule base, and ultimately generates an adjudication result.

[0028] The input terminal of the result output and management module 40 is connected to the output terminal of the system scheduling and fusion adjudication subsystem 30. The result output and management module 40 is used to receive adjudication results and convert them into a specified format for output or storage.

[0029] In one embodiment, the workflow of the automatic scoring system for pickleball is as follows: The data acquisition subsystem 10 continuously acquires video and audio stream data from the playing field and transmits it to the data processing subsystem 20. The data processing subsystem 20 performs preliminary processing on the audio stream data. Once a preset acoustic event is detected and located, the acoustic event information is transmitted to the system scheduling and fusion adjudication subsystem 30. Subsystem 30 then generates and sends back control commands to schedule the data processing subsystem 20 to perform refined analysis of the video stream data within a specified time window and spatial area, thereby extracting high-fidelity visual primitive information. Finally, the system scheduling and fusion adjudication subsystem 30 performs fusion analysis on the received acoustic event information and visual primitive information, generates an adjudication result, and transmits the adjudication result to the result output and management module 40 to complete a full event adjudication process.

[0030] The data acquisition subsystem 10 includes a multi-view visual acquisition module and a distributed acoustic acquisition module.

[0031] The multi-view visual acquisition module is used to simultaneously capture multiple high frame rate video streams covering the entire peak ball playing field.

[0032] The multi-view visual acquisition module includes multiple high-speed cameras. For example, it may include four high-speed cameras, deployed above and to the outside of the four corners of the Peak court. The field of view of the high-speed cameras is configured to ensure that their combined field of view can fully cover all areas of the entire playing field, including all boundary lines, non-volley zones, and the athletes' regular activity areas, and to create field-of-view overlap in key areas (such as near the non-volley zone lines).

[0033] The installation position and orientation of each high-speed camera are calibrated to obtain its precise extrinsic parameters in a preset world coordinate system, including the rotation matrix. Translation vector ,in This serves as an index for the cameras. The internal parameters of each high-speed camera are also calibrated to obtain its intrinsic parameter matrix. .

[0034] The high-speed cameras are equipped with high frame rate acquisition capabilities, with their frame rate set to a value sufficient to capture the details of the high-speed motion of the pickle, such as, but not limited to, 240 frames per second or higher. All high-speed cameras are configured for synchronous acquisition mode, meaning they acquire their respective image frames at the same time.

[0035] The choice of the acquisition frame rate was scientifically validated. Using the extreme case of a pickle's high-speed motion (e.g., a smash reaching 30 m / s) as a reference, at an acquisition rate of 240 frames / second, the maximum displacement of the ball between two frames is approximately 12.5 cm (30 m / s ÷ 240 frames / second). This displacement accuracy, combined with multi-view 3D reconstruction algorithms, is sufficient to accurately fit the ball's trajectory and meet the requirements for determining key events (such as whether the ball goes out of bounds or lands on a line), achieving a balance between acquisition efficiency and analysis accuracy.

[0036] All high-speed cameras are configured for synchronous acquisition mode, meaning they acquire their respective image frames at the same time.

[0037] To achieve synchronized acquisition, the multi-view visual acquisition module also includes a synchronization controller. The synchronization controller connects to all high-speed cameras and sends synchronization trigger signals to them. These trigger signals can be hardware trigger signals or software trigger commands issued via a precise time protocol, ensuring that the time error of image frames acquired by all high-speed cameras is within a preset, minimal threshold.

[0038] The multi-view visual acquisition module also includes a timestamp generation unit. This unit, associated with the synchronization controller, is used to attach a unified, high-precision timestamp to each set of image frames (i.e., frame groups) synchronously acquired by all high-speed cameras. This timestamp originates from a unified system clock source, which is also used to timestamp the audio data from the distributed acoustic acquisition module, thereby ensuring consistency in the time reference between subsequent acoustic and visual information.

[0039] The raw video stream data output by each high-speed camera, along with its corresponding timestamp, is transmitted to the data processing subsystem 20. The video stream data is stored in a circular buffer for subsequent routine processing or fine-grained retrospective analysis triggered by control commands by the data processing subsystem 20.

[0040] In one embodiment, a distributed acoustic acquisition module is used to simultaneously acquire multiple audio streams covering the entire peak ball playing area. This distributed acoustic acquisition module provides high-fidelity raw acoustic signals for subsequent acoustic event detection and spatiotemporal localization.

[0041] The distributed acoustic acquisition module includes multiple microphones. For example, it may include four or more microphones, which are deployed at predetermined locations around the perimeter of the playing field. The specific deployment locations are chosen to optimize the geometric accuracy factor of sound source localization; for example, the microphones can be mounted on the net posts on both sides of the field and on the fences or supports on both sides of the baseline, forming an array with good spatial envelope.

[0042] Each microphone is a high-sensitivity, wide-bandwidth condenser microphone or MEMS microphone, capable of capturing transient high-frequency signals generated by events such as the sound of a pickle hitting or landing. The three-dimensional physical position of each microphone in the world coordinate system is as follows: ; In the formula, the symbol Represents a three-dimensional column vector used to represent the first element in the distributed acoustic acquisition module. The physical spatial location of each microphone; symbol Representing the first A microphone in the world coordinate system The three Cartesian coordinate components in the equation; Represents a vector or matrix.

[0043] The three-dimensional physical positions in the world coordinate system are all precisely measured and calibrated and recorded by the system.

[0044] The distributed acoustic acquisition module also includes a multi-channel audio interface. All microphones are connected to this multi-channel audio interface. This multi-channel audio interface is responsible for synchronous analog-to-digital conversion of the analog audio signals from each microphone. The sampling rate of the analog-to-digital converter is set to a value sufficient to preserve high-frequency details of the signal, such as, but not limited to, 48kHz or 96kHz, and the sampling bit depth is, for example, 24 bits.

[0045] The sampling rate setting follows the Nyquist theorem. Considering that the target frequency band for subsequent bandpass filtering is 15kHz, a sampling rate of 48kHz (more than twice that of 15kHz) can ensure the distortion-free reconstruction of waveforms of key acoustic events and accurately capture the details of transient signals such as ball impacts, providing a high-quality signal foundation for subsequent accurate calculation of time delay differences.

[0046] To achieve synchronized acquisition across all channels, this multi-channel audio interface incorporates a shared sampling clock, ensuring that all audio channels are sampled at exactly the same time. This sampling clock is synchronized with the system clock source used by the multi-view vision acquisition module to achieve time alignment of acoustic and visual data streams at the acquisition level.

[0047] The distributed acoustic acquisition module also includes a timestamp appending unit associated with the multi-channel audio interface. This timestamp appending unit is responsible for appending a high-precision timestamp to each frame of synchronously acquired audio data blocks containing sampled data from all channels. This timestamp also originates from the aforementioned unified system clock source.

[0048] The multi-channel digital audio stream data, after synchronous acquisition, analog-to-digital conversion, and time-stamping, is output from the distributed acoustic acquisition module and transmitted to the data processing subsystem 20 for subsequent real-time signal processing and event analysis.

[0049] The data processing subsystem 20 includes an acoustic event processing module and a visual information processing module.

[0050] See attached document Figure 2 In one embodiment, the input of the acoustic event processing module is connected to the output of the distributed acoustic acquisition module, and its output is connected to the system scheduling and fusion adjudication subsystem 30. This acoustic event processing module is used to detect, classify, and locate key acoustic events in real time from multi-channel audio stream data, and output structured event information.

[0051] The acoustic event processing module includes a signal preprocessing unit, an event detection and classification unit, and a spatiotemporal positioning unit.

[0052] The signal preprocessing unit optimizes the received raw multi-channel digital audio stream data to enhance the target signal and suppress interference. This unit performs bandpass filtering on each audio channel. The passband frequency range of the filter is set to cover the main energy distribution frequency bands of the acoustic characteristics of key events such as the hitting and landing of the pickle. For example, it can be set between 1kHz and 15kHz to filter out low-frequency environmental noise (such as air conditioning noise) and high-frequency electronic noise in the venue.

[0053] The selection of this frequency range is based on spectral analysis of numerous key acoustic events in actual pickleball matches (including the sounds of hitting the ball with rackets of different materials and the sounds of the ball colliding with different court surfaces). Measured data shows that over 95% of the core energy of these key events is concentrated in the 1kHz to 15kHz frequency band. Therefore, this bandpass filter can effectively enhance the signal-to-noise ratio, retaining the vast majority of the effective signal while precisely filtering out interference noise unrelated to the match.

[0054] The event detection and classification unit is used to identify the occurrence of key events from the preprocessed audio stream data. This unit first calculates the short-time energy of the signal for each audio channel. When any channel's Exceeding a preset energy threshold within a time window When an event is detected, an event check is triggered.

[0055] After the event detection is triggered, the event detection and classification unit extracts a signal segment centered on the energy peak moment, containing all audio channels, to form a multi-channel audio sample. This multi-channel audio sample It is input into a pre-trained one-dimensional convolutional neural network classification model In the middle. This dimensional convolutional neural network classification model. The structure is designed to learn and distinguish acoustic fingerprints of different events from the time-domain and frequency-domain features of audio waveforms. (3D convolutional neural network classification model) The output is a label representing the event category. The set of category labels includes 'paddle-ball_impact' (paddle hitting the ball), 'ground-ball_impact' (ball landing), etc.

[0056] The spatiotemporal localization unit is used to calculate the precise occurrence time and three-dimensional spatial location of acoustic signals identified as critical events by the event detection and classification unit. For a classified critical event, the spatiotemporal localization unit first uses a cross-correlation algorithm or a generalized cross-correlation algorithm to calculate the estimated time delay difference between the arrival of the event signal at different microphones.

[0057] Suppose the event signal arrives at the th The microphone and the first The timing of each microphone is as follows: and Then the time delay difference between them is The spatiotemporal localization unit calculates the time delay difference vector of all other microphones with reference to one microphone (e.g., microphone 1). .

[0058] Subsequently, the spatiotemporal positioning unit uses these time delay differences and pre-calibrated three-dimensional positions of each microphone as a basis for positioning. Establish a location of the sound source The nonlinear equation system. Taking the microphone as a reference, the... The equations are: ; in, A mathematical expression for calculating the Euclidean distance between two points in three-dimensional space; Specifically refers to the distance between the sound source and the reference microphone; Represents the time difference of arrival; This is the speed at which sound travels through the air, and this speed value can be compensated for based on the ambient temperature. Location of the sound source coordinates For the first One microphone The coordinates.

[0059] The spatiotemporal localization unit employs numerical solution algorithms, such as the iterative least squares method based on Taylor series expansion, to solve the system of equations and obtain the sound source location. The estimated value.

[0060] In obtaining the location of the sound source Afterwards, the precise timestamp of the event. It is calculated using the following formula: ; in, It is the moment when the reference microphone detects the peak of the signal energy; For reference microphone position; Denotes the Euclidean norm; The straight-line distance between the sound source location and the reference microphone location.

[0061] Finally, the acoustic event processing module will output a structured acoustic event information packet, which includes the event category. Event timestamp and the location of the sound source of the event The acoustic event information packet was transmitted to the system scheduling and fusion adjudication subsystem 30.

[0062] See attached document Figure 3 In one embodiment, the input terminal of the visual information processing module is connected to the output terminal of the multi-view visual acquisition module, and its output terminal is connected to the system scheduling and fusion adjudication subsystem 30. Simultaneously, the visual information processing module is provided with a control input terminal for receiving control commands from the system scheduling and fusion adjudication subsystem 30.

[0063] The visual information processing module analyzes multi-channel video stream data to extract visual primitive information relevant to the competition. This module features a dual-modal operating mechanism, comprising a conventional monitoring unit and a refined analysis unit. The module switches between these two operating modes based on instructions received from the control input.

[0064] The routine monitoring unit is activated when the system is in its default baseline operating state. The purpose of this routine monitoring unit is to maintain basic situational awareness of the main dynamic objects on the playing field (i.e., players and the picket ball) with low computational resource consumption.

[0065] In this mode, the conventional monitoring unit downsamples the high frame rate video stream received from the multi-view visual acquisition module, for example, reducing the temporal frame rate from 240 frames per second to 30 frames per second and the spatial image resolution. Subsequently, the conventional monitoring unit runs a lightweight target detection model on the downsampled video frames. This target detection model is used to identify and output the bounding box positions of players and the ball in real time in the two-dimensional image of each viewpoint. The conventional monitoring unit only outputs coarse two-dimensional position information of these targets, or no information is output at all; its main function is to ensure that the system is in a low-power standby state when no critical events occur.

[0066] The refined analysis unit is activated after the visual information processing module receives a refocusing control command from the system scheduling and fusion adjudication subsystem 30. This control command specifies a specific time window. It also includes a three-dimensional spatial region of interest (ROI). This refined analysis unit is used for high-precision, computationally intensive analysis of data within a specified spatiotemporal range.

[0067] Once activated, the fine-grained analysis unit first extracts the full-resolution, high-frame-rate original image frame sequence of all channels within the spatiotemporal range specified by the control command from the circular buffer storing multi-channel video stream data.

[0068] Subsequently, the refinement analysis unit performs multiple high-precision visual analysis tasks in parallel on the extracted image frame sequence: Multiview Figure 3 2D Human Pose Estimation: This refined analysis unit first runs a high-precision 2D human keypoint detection model on the player within the ROI in the 2D image from each viewpoint to obtain their full-body image. The two-dimensional coordinates of key points (such as joints and facial features) are then obtained. The refined analysis unit then uses calibrated multi-camera intrinsic and extrinsic parameters, through triangulation or an optimized multi-view fusion algorithm, to reconstruct the player's three-dimensional skeletal key point set in the world coordinate system. ; in, Represents a set that fully describes the first... Athletes at precise timestamps Three-dimensional human posture at that time; Indicates belonging to the three-dimensional real space ; Indicates index It will start from 1 and take all integers in turn until... ; It is a positive integer constant representing the total number of keypoints contained in the human skeleton model used.

[0069] Precise Sphere Segmentation and 3D Reconstruction: This refined analysis unit runs an instance segmentation model for a pickle within the ROI in each viewpoint image. The model outputs a pixel-level precise mask of the sphere in the image. By performing spatial intersection operations or visual shell reconstruction on the sphere contours (defined by the mask edges) from multiple viewpoints, this refined analysis unit can calculate the high-precision 3D center position of the sphere in the world coordinate system. .

[0070] The refined analysis unit will extract high-fidelity visual primitive information, including but not limited to the player's three-dimensional pose sequence. and the three-dimensional trajectory sequence of the sphere The data is structured and encapsulated, and then transmitted to the system scheduling and fusion adjudication subsystem 30 for subsequent information fusion and event reasoning. After completing a refined analysis task, the refined analysis unit automatically switches to an inactive state, and the visual information processing module reverts to the conventional monitoring mode dominated by the conventional monitoring unit.

[0071] The system scheduling and fusion decision subsystem 30 includes a dynamic refocusing control module 31 and a multimodal information fusion module.

[0072] See attached document Figure 5 In one embodiment, the information input terminal of the dynamic refocusing control module 31 is connected to the output terminal of the acoustic event processing module, and the control output terminal is connected to the control input terminal of the visual information processing module. The dynamic refocusing control module 31 generates and sends a control command based on the received acoustic event information. This command switches the working mode of the visual information processing module from the conventional monitoring mode to the refined analysis mode, and specifies the time and spatial range to be focused on during the analysis.

[0073] The dynamic refocusing control module 31 receives a structured acoustic event information packet from the acoustic event processing module. This information packet contains event categories. Event timestamp and the location of the sound source of the event Location of the sound source These are coordinates in the world coordinate system.

[0074] The dynamic refocusing control module 31 includes an event filtering unit. This event filtering unit first categorizes the events in the received acoustic event information packets. Make a judgment. Only if the event category... The dynamic refocusing control module 31 only initiates the subsequent control command generation process when the event belongs to a preset set of key event categories. If the event category does not belong to the set, the acoustic event information packet is discarded and no control command is generated.

[0075] After passing the event filtering, the dynamic refocusing control module 31 generates a refocusing control command. This instruction It is a data structure that consists of the following two parts: A time window This time window is based on the timestamp of the acoustic event. Centered on, it is defined as It is a pre-defined system parameter that represents the time span of retrospective and prospective analysis.

[0076] A three-dimensional spatial region of interest The region of interest is defined by the location of the sound source of the acoustic event. Centered on, it is defined as a geometric body with a preset size, for example, a body with a side length of... A cube. Among them, It is a pre-defined system parameter that represents the scope of spatial analysis.

[0077] Control commands After generation, the dynamic refocusing control module 31 sends it to the control input of the visual information processing module through the control output terminal.

[0078] The visual information processing module receives the control command at its control input terminal. Subsequently, its internal control logic is triggered, causing a switch in its operating state: the activity of its regular monitoring unit is paused, while its refined analysis unit is activated. Furthermore, the refined analysis unit strictly limits its subsequent data extraction and analysis operations to those initiated by control commands. The specified spatiotemporal range, i.e., the time window and spatial interest areas Within.

[0079] See attached document Figure 4 In one embodiment, the input of the multimodal information fusion module is connected to the outputs of the acoustic event processing module and the visual information processing module, and its output is connected to the result output and management module 40. This multimodal information fusion module integrates data from different modalities and performs reasoning based on preset rules to generate the final match decision.

[0080] The multimodal information fusion module includes a causal event graph construction unit and a rule reasoning and adjudication unit.

[0081] The causal event graph construction unit is used to organize received discrete acoustic event information and visual primitive information into a dynamically updated data structure with spatiotemporal and logical relationships. This data structure is defined as a directed acyclic graph. .

[0082] Directed acyclic graph Node set Includes two types of nodes: Acoustic event nodes Each acoustic event node This corresponds to an acoustic event output by the acoustic event processing module. The node's attributes include: a unique identifier, and the event category. Event timestamp and the location of the sound source of the event .

[0083] Visual primitive nodes Each visual primitive node This corresponds to a high-fidelity visual state extracted by the refined analysis unit of the visual information processing module. The node's attributes include: a unique identifier, a primitive category, and a timestamp. and specific data loads (e.g., a set of 3D skeletal keypoints of a player). Or the three-dimensional center position of the sphere ).

[0084] Directed acyclic graph edge set There are two types of edges used to represent the relationships between nodes: Timing edge This type of edge is used to connect visual primitive nodes representing the same physical object (e.g., the same player or pickle) at consecutive timestamps. For example, the sphere at timestamps... and The two visual primitive nodes are connected by a temporal edge, and these edges together constitute the three-dimensional motion trajectory of the object.

[0085] Associated edges This type of edge is used to connect different types of nodes that are strongly correlated in space and time. When an acoustic event node... timestamp Harmony and sound source location With one or more visual primitive nodes timestamp When the location data matches within a preset tolerance range, the causal event graph construction unit establishes a correlation edge between them.

[0086] The preset tolerances here are rigorously set to ensure the reliability of the fusion. The time tolerance (e.g., ±50 milliseconds) primarily considers the maximum synchronization error that may exist between data streams from different sensors within the system, the computational delay introduced by the signal processing flow, and the duration of the acoustic event itself. The spatial tolerance (e.g., ±15 centimeters) combines the inherent errors of the acoustic localization algorithm (depending on the geometry of the microphone array and environmental reflections) with the accuracy errors of multi-view visual 3D reconstruction. This tolerance range is set to ensure that multimodal data points generated by the same physical event can be correctly correlated, while avoiding the incorrect connection of unrelated events. Its value is based on the comprehensive error statistics in the system calibration test, rather than being arbitrarily set.

[0087] For example, an acoustic event node of the category 'paddle-ball_impact' will be associated with player posture nodes and ball state nodes that occur in similar time and space.

[0088] The rule-based reasoning and adjudication unit is used in the directed acyclic graph generated by the causal event graph construction unit. The system performs logical queries and traversals to determine whether a point, mistake, or foul has occurred. This rule reasoning and adjudication unit pre-stores a rule knowledge base for pickleball, where each rule is encoded as one or more rules targeting a directed acyclic graph. The structured query process.

[0089] Taking the determination of a non-volleyball foul as an example, the query process includes the following steps: In a directed acyclic graph Query a node The event category for this node is 'paddle-ball_impact'.

[0090] Through with The associated sphere visual primitive node is used to trace the sphere's movement trajectory in the current round (i.e., along the temporal edge). (Reverse traversal). On this trajectory, check if any acoustic event node of category 'ground-ball_impact' exists. If not, then record this ball-hitting event. Marked as a volley.

[0091] If the shot is marked as a volley, then obtain the same information as the shot. Related player posture visual primitive nodes .

[0092] from Extract the player's hit timestamp from the data load. The coordinates of key points of the foot's three-dimensional skeleton at that time.

[0093] The coordinates of key points on the foot are compared with a pre-calibrated three-dimensional geometric model of the non-volley zone. A geometric judgment algorithm based on the point in the polyhedron is used to determine whether the player's foot was inside the non-volley zone or touched its boundary at the moment of impact.

[0094] If the marking result is a volley and the judgment result is true, the rule reasoning and adjudication unit generates an adjudication result of a non-volley zone foul.

[0095] The rule reasoning and adjudication unit will structure and encapsulate the generated adjudication results (including event type, adjudication conclusion, timestamp, players involved, etc.) and transmit them to the result output and management module 40.

[0096] See attached document Figure 1 The input terminal of the result output and management module 40 is connected to the output terminal of the system scheduling and fusion adjudication subsystem 30. The result output and management module 40 is used to receive and process the final match adjudication results, and present and record them in a specified format.

[0097] In one embodiment, the results output and management module 40 includes a real-time data display unit and a match report generation unit.

[0098] The real-time data display unit is used to present the adjudication results generated by the system scheduling and fusion adjudication subsystem 30 to the athletes, referees, and spectators on site in real time and visually. The input of the real-time data display unit receives structured adjudication result information, which includes at least the adjudication conclusion (e.g., points scored, points conceded, fouls), the players involved, and the current total score.

[0099] The real-time data display unit includes a communication interface for converting received decision result information into a data format conforming to a specific display device protocol. For example, this communication interface could be an HDMI (High-Definition Multimedia Interface) output interface, an Ethernet interface, or a wireless communication module.

[0100] The real-time data display unit is connected to one or more external display devices, such as a large LED (light-emitting diode) display screen deployed next to the playing field. When a new ruling is received, the real-time data display unit immediately updates the display content, for example, updating the score on the screen, or displaying the type of foul and information about the offending party when a foul is determined.

[0101] The competition report generation unit is used to persistently store all key events and decisions throughout the competition for post-competition querying, replay, and data analysis. The input to this unit also receives decision results from the system scheduling and fusion decision subsystem 30.

[0102] The competition report generation unit contains a structured database. Whenever a ruling is received, the unit not only records the ruling itself, but also requests and retrieves the directed acyclic graph associated with that ruling from the system scheduling and fusion ruling subsystem 30. This is a data snapshot. The snapshot contains detailed attributes of all acoustic event nodes and visual primitive nodes upon which this ruling is based.

[0103] The competition report generation unit stores the ruling conclusion, the ruling timestamp, and the corresponding causal event graph data snapshot as a complete entry in the database. In this way, each ruling has complete data traceability.

[0104] After the match, or upon user instruction, the match report generation unit can extract all records from the database for a specified time period or the entire match, and automatically generate a detailed match report according to a preset template. This report may include, but is not limited to: the final score of both teams, a detailed breakdown of each point scored and lost, a list of all fouls and their timing and basis, and other key match statistics. The report can be output as an electronic document (e.g., PDF) or provide an interactive data query interface.

[0105] It is worth emphasizing that several core innovations of this invention, distinguishing it from existing technologies, have been implemented through key algorithmic steps in the specific technical solution and constitute an important part of the scope of protection of this invention. Specifically, in the acoustic processing section, the spatiotemporal positioning unit employs a technique based on cross-correlation or generalized cross-correlation algorithms to calculate the time delay difference between signals arriving at different microphones, which is the foundation for achieving high-precision, non-contact sound source localization. In the visual processing and system scheduling section, the mechanism by which the refined analysis unit extracts the full-resolution, high-frame-rate original image frame sequence within the spatiotemporal range specified by control commands from the circular buffer is the core of achieving dynamic refocusing and significantly reducing the average power consumption of the system while ensuring analysis accuracy. These two key algorithmic steps ensure the feasibility and advancement of the technical solution of this invention and have been defined in the corresponding claims.

[0106] See attached document Figure 1 -Appendix Figure 5 The following will take the judgment process of a non-volleyball foul as an example to explain in detail the collaborative workflow of the automatic scoring system for pickball provided by this invention.

[0107] First, the automatic scoring system for the pickleball game is in its initial operational state. The data acquisition subsystem 10 operates continuously, with the multi-view visual acquisition module capturing multiple video streams and the distributed acoustic acquisition module acquiring multiple audio streams. The data processing subsystem 20 receives these data streams, and its internal visual information processing module operates in conventional monitoring mode, where the conventional monitoring unit performs low-power, coarse target tracking on the video streams.

[0108] During a match, a player volleyed the ball near the non-volley line. The impact sound waves from this shot were captured by multiple microphones in the distributed acoustic acquisition module.

[0109] The captured multi-channel audio stream data is transmitted to the acoustic event processing module. The signal preprocessing unit of this module filters and reduces noise in the signal. Subsequently, the event detection and classification unit detects energy pulses in the signal and classifies them using a one-dimensional convolutional neural network model. The event is identified as belonging to the 'paddle-ball_impact' category.

[0110] Subsequently, the spatiotemporal positioning unit calculates the three-dimensional spatial location of the ball-hitting event based on the time delay difference between the arrival of the event signal at different microphones. and precise timestamp of occurrence The acoustic event processing module then generates a structured acoustic event information packet containing the event category, location, and timestamp, and sends it to the system scheduling and fusion adjudication subsystem 30.

[0111] The dynamic refocusing control module 31 in the system scheduling and fusion adjudication subsystem 30 receives the information packet. After confirming that the event category is a critical event, the dynamic refocusing control module 31 immediately generates a refocusing control command. This refocusing control command Specify timestamp Centered time window and the location of the sound source Spatial interest area centered .

[0112] This control command The data is sent to the control input of the visual information processing module. The visual information processing module immediately switches its operating mode, and its refined analysis unit is activated. This refined analysis unit retrieves data from the circular buffer... The specified full-resolution, high-frame-rate image frame sequence is used, and subsequent visual algorithm processing resources are concentrated on the [processor name missing]. Within the defined spatial region.

[0113] On the specified spatiotemporal data, the refined analysis unit performs high-precision analysis: it reconstructs the batter's position at the timestamp. The continuous 3D pose of the nearby object was captured, specifically the 3D coordinates of its foot keypoints; simultaneously, the 3D trajectory of the pick ball within that time window was reconstructed. This extracted high-fidelity visual primitive information was then transmitted to the multimodal information fusion module.

[0114] The causal event graph construction unit in the multimodal information fusion module receives acoustic event information packets and visual primitive information. It operates within a dynamic directed acyclic graph. Create an acoustic event node representing the current shot, and a series of visual primitive nodes representing the player's posture and the ball's position. Establish association edges and temporal edges between these nodes based on spatiotemporal proximity.

[0115] Subsequently, the rule-based reasoning and adjudication unit updates the directed acyclic graph. The system executes a pre-defined NVZ foul determination process. It first confirms the shot as a volley by tracing the ball's trajectory, then retrieves the player's posture nodes associated with the shot event and compares the 3D coordinates of the feet with a pre-stored 3D model of the non-volley zone. The determination is that the player's foot touched the boundary of the non-volley zone at the moment of impact.

[0116] Based on the above determination, the rule reasoning and adjudication unit generates a final adjudication result for a non-volley zone hitting foul and transmits the result to the result output and management module 40.

[0117] Finally, the real-time data display unit of the results output and management module 40 sends the foul information and the updated score to the LED display screen on the sidelines. Simultaneously, the match report generation unit stores the ruling result, along with a snapshot of the causal event graph data on which it is based, into the database. After completing this ruling, the visual information processing module automatically reverts to normal monitoring mode, awaiting the next critical event.

Claims

1. An automatic scoring system for pickleball based on multi-view high-speed video recording and AI recognition, characterized in that: include: The data acquisition subsystem is used to acquire multimodal raw data in real time from the pickleball court, the multimodal raw data including multi-channel synchronized video stream data and multi-channel synchronized audio stream data; A data processing subsystem is used to receive the multimodal raw data, analyze and process the multimodal raw data, and extract structured acoustic event information and visual primitive information. The system scheduling and fusion adjudication subsystem is used to generate control commands based on the acoustic event information output by the data processing subsystem and send them to the data processing subsystem. It is used to dynamically adjust the working mode and resource allocation of the data processing subsystem. It is also used to fuse the acoustic event information and the visual primitive information, perform logical reasoning based on the preset competition rule library, and finally generate the adjudication result. The result output and management module is used to receive the adjudication results and convert them into a specified format for output and storage.

2. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition as described in claim 1, characterized in that, The data acquisition subsystem includes: A multi-view visual acquisition module is used to simultaneously capture multiple high-frame-rate video streams covering the entire peak ball competition field and generate multiple synchronized video streams. The multi-view visual acquisition module includes multiple high-speed cameras, a synchronization controller, and a timestamp generation unit. A distributed acoustic acquisition module is used to synchronously acquire multiple audio streams covering the entire peak ball playing field and generate multiple synchronous audio streams. The distributed acoustic acquisition module includes multiple microphones, a multi-channel audio interface, and a timestamp attachment unit.

3. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition as described in claim 1, characterized in that, The data processing subsystem includes: An acoustic event processing module is used to detect, classify and locate key acoustic events in real time from the multi-channel synchronized audio stream data in the multimodal raw data, and output structured acoustic event information. The visual information processing module is used to analyze the multi-channel synchronized video stream data in the multimodal raw data and extract the visual primitive information.

4. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition according to claim 3, characterized in that, The acoustic event processing module includes: The signal preprocessing unit is used to perform bandpass filtering preprocessing on the multi-channel synchronized audio stream data in the received multimodal raw data; The event detection and classification unit is used to perform short-time energy calculation on the preprocessed multi-channel synchronized audio stream data to trigger event detection, and input it into a pre-trained 3D convolutional neural network classification model to identify the event category; The spatiotemporal positioning unit uses a cross-correlation algorithm or a generalized cross-correlation algorithm to calculate the estimated time delay difference between signals arriving at different microphones. Based on the time delay difference and the pre-calibrated three-dimensional positions of each microphone, it establishes and solves a set of nonlinear equations to calculate the sound source location and the precise occurrence timestamp, which is used to generate acoustic event information.

5. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition according to claim 3, characterized in that, The visual information processing module has a dual-modal working mechanism, specifically including: The conventional monitoring unit is activated when the system is in the default baseline working state. It is used to downsample the multi-channel synchronous video stream data in the multimodal raw data and run a lightweight target detection model to obtain the rough two-dimensional position information of the player and the ball. The refined analysis unit is activated after receiving a refocusing control command from the system scheduling and fusion adjudication subsystem. It is used to extract the full-resolution, high-frame-rate original image frame sequence within the spatiotemporal range specified by the control command from the circular buffer, and to perform multi-view 3D human pose estimation and precise sphere segmentation and 3D reconstruction to extract the visual primitive information.

6. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition according to claim 1, characterized in that, The system scheduling and fusion adjudication subsystem includes: The dynamic refocusing control module is used to generate and send the control command based on the received acoustic event information. The control command switches the working mode of the visual information processing module from the conventional monitoring unit to the refined analysis unit and specifies the time and spatial range that the analysis should focus on. The multimodal information fusion module is used to integrate acoustic event information with the visual primitive information, perform logical reasoning based on a preset competition rule library, and generate the final competition ruling result.

7. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition according to claim 6, characterized in that, The dynamic refocusing control module includes: An event filtering unit is used to determine the event category in the acoustic event information. When the event category belongs to a preset set of key event categories, the control command generation process is initiated. The control command comprises the following two data structures: A time window, centered on the timestamp of the event in the acoustic event information, and including a preset time span; A three-dimensional spatial region of interest, which is centered on the three-dimensional position of the event in the acoustic event information and defined as a geometric body with a preset size.

8. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition according to claim 1, characterized in that, The multimodal information fusion module includes: A causal event graph construction unit is used to organize the received discrete acoustic event information and visual primitive information into a dynamically updated directed acyclic graph with spatiotemporal and logical relationships. The node set of the directed acyclic graph includes acoustic event nodes and visual primitive nodes, and the edge set of the directed acyclic graph includes temporal edges and associative edges. The rule reasoning and adjudication unit is used to perform logical queries and traversals on the directed acyclic graph to determine whether a score, mistake, or foul has occurred.

9. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition according to claim 8, characterized in that, In the causal event graph construction unit: The acoustic event node corresponds to an acoustic event information output by the acoustic event processing module, and its attributes include a unique identifier, event category, event occurrence timestamp, and event three-dimensional location; The visual primitive node corresponds to the visual primitive information extracted by the refined analysis unit of the visual information processing module, and its attributes include a unique identifier, primitive category, timestamp, and specific data load. The temporal edge is used to connect the visual primitive nodes representing the same physical object at consecutive timestamps; The associated edge is used to connect different types of nodes that are strongly correlated in time and space. The associated edge is established when the timestamp and location of an acoustic event node match the timestamp and location data of one or more visual primitive nodes within a preset tolerance range.

10. The automatic scoring system for pickleball based on multi-view high-speed camera and AI recognition according to claim 8, characterized in that, The result output and management module includes: A real-time data display unit is used to present the ruling results to the athletes, referees, and spectators on site in real time and visually. The match report generation unit is used to persistently store all key events and rulings throughout the match. It has an internal structured database. Whenever a ruling is received, the match report generation unit records the ruling conclusion, the ruling timestamp, and a data snapshot of the causal event graph associated with this ruling.

Images