Multi-modal sensing and AI algorithm-based sports competition real-time penalty system and method

Abstract

The invention relates to the technical field of sports competition real-time penalty, and discloses a sports competition real-time penalty system and method based on multi-modal sensing and AI algorithms, and the system comprises a monitoring identification module, an instrument tracking module, a rule parameter construction module, a fusion penalty module, an output and snapshot module, and a wireless linkage and energy management module. The method comprises the following steps: acquiring image and instrument data, and constructing a multi-modal sensing structure; calling a rule base to generate a penalty vector and loading a structure constraint; fusing the image and the instrument features to generate a feature tensor; inputting an AI model to judge the legality of the action and outputting a penalty result; driving the snapshot logic to generate an image record and synchronizing the image record to the terminal; and adjusting the energy consumption strategy according to the task load and the equipment state. According to the invention, a multi-modal perception input structure fusing instrument motion data and a visual image flow is introduced, so that the system can synchronously acquire displacement, acceleration and attitude images in the action recognition process, and a penalty basis is checked in a cross manner from multiple dimensions.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time sports competition penalty judgment, and specifically to a real-time sports competition penalty judgment system and method based on multimodal sensing and AI algorithm. Background Art

[0002] With the rapid development of artificial intelligence and sensing technologies, the application of intelligent assistance systems in sports is becoming increasingly widespread. In particular, in competitive sports and skill assessment events, accurately and efficiently identifying and judging athletes' technical movements, violations, and game status has become a core technical requirement for improving referee efficiency and fairness. At the same time, the demand for "visualized penalty evidence" and "automated system response" in various competitions is constantly increasing, prompting an urgent trend to upgrade traditional referee assistance equipment to intelligent penalty systems.

[0003] Some existing sports competitions have introduced video-assisted systems based on image recognition technology to detect obvious violations such as athlete crossing the line, touching the line, and premature starting. A common system architecture uses multiple high-definition surveillance cameras to capture live game footage, combined with an edge computing platform to run motion recognition algorithms to achieve preliminary penalty decisions.

[0004] However, existing penalty judgments are based on a single, unreliable basis and lack three-dimensionality. They rely on camera images, resulting in unstable recognition results in scenarios with occlusion, complex movements, or high-frequency switching. Furthermore, the rule judgment model is fixed and cannot be flexibly adapted to different events or action groups. The expressive power of the rules is limited, and image capture and penalty results are often decoupled, making it difficult to automatically backtrack and form visual evidence. Furthermore, the linkage terminal control and energy consumption management mechanisms are extensive, which can easily lead to response delays and resource waste. Therefore, the present invention provides a real-time sports competition penalty judgment system and method using multimodal sensing and AI algorithms to address the shortcomings of the existing technology. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a real-time sports competition penalty system and method based on multimodal sensing and AI algorithm, which solves the problems of existing real-time sports competition penalty technology in terms of single penalty dimension, rigid rules that are difficult to adapt, low image capture synchronization accuracy, and rough terminal response and energy consumption management mechanism.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a real-time sports competition penalty judging system based on multimodal sensing and AI algorithm, comprising: The monitoring recognition module is used to perform human figure recognition and motion tracking based on image data collected by monitoring equipment, generating target detection results and posture timing information; The equipment tracking module includes a tracking chip installed on the sports equipment, which is used to collect the equipment's position information, velocity data, and acceleration parameters through the tracking chip, and synchronize the equipment motion data with the posture timing information to build a multimodal perception input structure; A rule parameter construction module is used to receive the preset sports game rule configuration, generate a penalty rule vector that matches the action category and violation conditions, and provide structural constraints for AI penalty decision-making; The fusion penalty module is used to construct a fusion feature tensor based on the multimodal perception input structure and the penalty rule vector, and input it into the AI ​​model for action compliance classification to generate penalty results and confidence information; The output and capture module is used to drive the image capture logic according to the penalty results, generate penalty image records of the corresponding frames, and synchronize the penalty results and image data to the electronic display screen and user terminal; The wireless linkage and energy management module is used to establish a low-latency wireless communication link, monitor the power status of the real-time penalty system, dynamically adjust the power consumption strategy according to the perceived task intensity, manage the device charging status and control the system linkage.

[0007] Preferably, the monitoring and identification module includes: The target detection unit is used to detect human targets in image data, extract the athlete's contour information and action key points, and perform target positioning and recognition based on convolutional neural networks; The motion tracking unit is used to track the athlete's motion changes through a time-series tracking algorithm, generate motion trajectories and posture change time series, and establish the athlete's real-time motion model; The motion analysis unit is used to classify and analyze the detected movements, generate specific labels for each movement based on historical data, and identify whether the movement complies with the competition rules.

[0008] Preferably, the performing human target detection on the image data includes: Athlete position coordinates, representing the athlete's position in the competition venue, are calculated as follows: ; in, Indicates the current position of the athlete. are the image coordinates of the athlete, is the acceleration data; Action key points, using deep learning methods to identify the key points of athletes and map them to the field coordinate system; The posture estimation results estimate the posture of each part of the athlete's body based on the athlete's image data, including the angle information of the torso and limbs.

[0009] Preferably, the device tracking module includes: Tracking chip units are installed on various devices and use radio frequency identification technology to collect the device's location information and movement status in real time; The speed and acceleration measurement unit is used to record the movement speed and acceleration data of the instrument, and capture the acceleration changes of the instrument at different time points through sensors; The data synchronization unit is used to synchronize the position information of the equipment with the athlete's posture timing information.

[0010] Preferably, the rule parameter construction module includes: A rule input unit for receiving action determination rules configured by the competition organizer, including specific action standards and violation conditions for different sports; A penalty rule vector generation unit generates a penalty rule vector that complies with the competition specifications based on the received rule information and stores it in a data structured manner; The structural constraint generation unit combines real-time action data with competition rules to generate constraints that need to be followed during the penalty process, including action legality, time window and speed limit.

[0011] Preferably, the fusion penalty module includes: The feature fusion unit is used to merge the data from the monitoring and recognition module and the device tracking module, and generate a feature fusion tensor through the weighted average method: ; in, Indicates the comprehensive penalty result. To monitor data characteristics, is the device data feature, and is the weight; The action compliance classification unit is used to judge the compliance of the action based on the fused feature tensor and output the action legality probability and penalty level; The confidence calculation unit is used to calculate the confidence of the penalty result and output the penalty result and its credibility index.

[0012] Preferably, the output and capture module includes: An image driving unit, configured to generate a corresponding image capture instruction according to the penalty result output by the fusion penalty module; An image acquisition trigger unit, configured to control the monitoring camera device to capture an image at a specific time point based on the capture instruction; The image upload and presentation unit is used to synchronously transmit the captured image data and penalty label information to the terminal presentation screen or the user's visual terminal.

[0013] Preferably, the wireless linkage and energy management module includes: A linkage instruction generation unit, configured to generate a trigger instruction for controlling a linkage response of a terminal device based on a system penalty result or sensory feedback; a wireless communication unit, configured to send the trigger instruction to a target device via Bluetooth, Wi-Fi, or other short-range wireless communication protocols; The energy status monitoring unit is used to monitor and adjust the energy supply status of the terminal device in real time, automatically enter low-power mode when the device is in standby mode, and quickly wake up to respond to control when receiving linkage tasks.

[0014] Preferably, the penalty result is calculated by a confidence function: ; in, For the The probability value of the class action, The reliability of the final penalty result, The total number of action categories.

[0015] A real-time sports competition penalty judgment method based on multimodal sensing and AI algorithm is also provided, which includes the following steps: The monitoring equipment collects image data including athlete images and postures, and the tracking chip collects equipment data. The image data and equipment data are processed synchronously to build a multimodal perception input structure that includes posture timing and equipment dynamics. Based on the action categories, motion trajectories, and equipment features in the multimodal perception input structure, the system calls the preset rule library for sports events, generates penalty rule vectors related to the detected actions, and dynamically loads structural constraints. Based on the multimodal input structure and penalty rule vector, the image features and device features are aligned and weighted fused to generate a fused feature tensor; The fused feature tensor is input into the trained AI penalty model to perform action compliance classification calculations and generate penalty output results including action category labels, violation determination results, and confidence values; Drive the image capture logic based on the penalty results, extract the video frame at the corresponding time point, generate the penalty image record, and synchronize the penalty image record and penalty label to the on-site electronic display screen and user terminal; Based on the task load and operating status of the penalty system, the power level of each device is monitored in real time, and the power consumption adjustment strategy is set according to the task intensity.

[0016] The present invention provides a real-time sports competition penalty judging system and method based on multimodal sensing and AI algorithms. It has the following beneficial effects: 1. This invention introduces a "multimodal perception input structure" that fuses machine motion data with visual image streams. This allows the system to simultaneously collect displacement, acceleration, and posture images during the motion recognition process, cross-checking the judgment basis from multiple dimensions. Previous approaches that relied solely on image analysis were prone to misjudgment in scenes with occlusion and blur. This structure overcomes the recognition distortion caused by single-source perception and maintains stable judgment even in complex motion environments.

[0017] 2. This invention generates a compliance score based on a comparison of dynamic penalty rule vectors with real-time action streams, and outputs penalty judgments in conjunction with a confidence mechanism. This completely eliminates the bottleneck of the old "hard-coded rules + manual review" system. Instead of rigid matching, it offers a flexible and adaptable structure that adapts to multiple events and action types, addressing the lack of dynamic adaptability of traditional rules and significantly improving the system's universality and scalability.

[0018] 3. During the output and capture phase, this invention uses a reverse time-window registration method to lock the key frames of the illegal action, structurally binding them to the penalty label and transmitting them back synchronously, significantly improving the consistency between image evidence and behavioral judgment. Previous systems generally relied on manual backtracking, resulting in delayed capture and inaccurate positioning. This solution automatically identifies the optimal image segment based on the time window, resolving issues such as ambiguous judgment criteria and image time sequence misalignment, and improving referee review efficiency.

[0019] 4. This invention uses state awareness and energy consumption classification strategies to achieve intelligent device wake-up and sleep switching. This eliminates the need for devices to remain in high-power standby mode. Existing tournament terminals suffer from redundant power consumption and severe response delays. This solution achieves a balanced operation with high responsiveness and low energy consumption, significantly improving operational stability in field deployments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a system architecture diagram of the present invention; Figure 2 The figure is a flow chart of the method steps of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] Please see the attached Figure 1 The embodiment of the present invention provides a real-time sports competition penalty judging system using multimodal sensing and AI algorithms, including: The monitoring recognition module is used to perform human figure recognition and motion tracking based on image data collected by monitoring equipment, generating target detection results and posture timing information; The equipment tracking module includes a tracking chip installed on the sports equipment, which is used to collect the equipment's position information, velocity data, and acceleration parameters through the tracking chip, and synchronize the equipment motion data with the posture timing information to build a multimodal perception input structure; A rule parameter construction module is used to receive the preset sports game rule configuration, generate a penalty rule vector that matches the action category and violation conditions, and provide structural constraints for AI penalty decision-making; The fusion penalty module is used to construct a fusion feature tensor based on the multimodal perception input structure and the penalty rule vector, and input it into the AI ​​model for action compliance classification to generate penalty results and confidence information; The output and capture module is used to drive the image capture logic according to the penalty results, generate penalty image records of the corresponding frames, and synchronize the penalty results and image data to the electronic display screen and user terminal; The wireless linkage and energy management module is used to establish a low-latency wireless communication link, monitor the power status of the real-time penalty system, dynamically adjust the power consumption strategy according to the perceived task intensity, manage the device charging status and control the system linkage.

[0023] For the monitoring and recognition module, in this embodiment, to achieve real-time behavior recognition and motion tracking of athletes during sports competitions, a multimodal data acquisition mechanism driven by image perception is required. This module not only recognizes humanoid targets but also performs temporal modeling of posture changes during dynamic processes. This serves as the image input foundation for the "human-object" coupled judgment of the entire system.

[0024] The monitoring and identification module is set on the monitoring and acquisition equipment at different angles of the competition venue. It usually adopts a fixed or pan-tilt high-definition camera system, and completes on-site identification through an image processing unit and an edge computing terminal.

[0025] Generally speaking, this module may include three submodules: target detection unit, motion tracking unit and motion analysis unit.

[0026] Specifically, the object detection unit locates human targets in the input image data, identifying the athlete's outline and defining their key structures. A common implementation approach is to build a detection framework based on a convolutional neural network (CNN) or a Transformer-style multi-head attention model. Typical models include YOLOv8 and CenterNet.

[0027] In one possible implementation, to enhance the accuracy of keypoint detection in images, the system introduces a human pose recognition network after object detection to further extract the coordinates of multiple keypoints, including the head, shoulders, elbows, hips, knees, and ankles. Keypoint extraction can be based on multi-stage regression frameworks such as HRNet and OpenPose.

[0028] To enhance the ability to model different postures during image recognition, the system introduces a time-series-based posture evolution tracking mechanism. In this embodiment, the motion tracking unit uses a sequential tracking method, such as an LSTM architecture with an attention mechanism, to sequentially model the motion states between adjacent frames and generate a time series of posture changes.

[0029] In some embodiments, to avoid missing key points due to image blur or occlusion, the motion tracking unit incorporates multi-frame fusion technology. Using a windowed weighted average mechanism, the pose results of several consecutive frames are structurally matched and compensated for missing key points, ensuring temporal consistency.

[0030] In this embodiment, after completing key point recognition and posture estimation, the physical position coordinates of the athlete in the field are further calculated. Specifically, position mapping is completed through the following function: ; in, Represents the physical position of the athlete in the field space coordinate system; is the target detection result of the athlete in the image coordinate system of the current frame, in the form of a two-dimensional vector group; is the real-time acceleration vector from the device tracking module or other auxiliary sensors. This function combines field calibration data with affine mapping to complete coordinate conversion.

[0031] As an option, the system can also introduce perspective transformation and monocular depth estimation modules to estimate depth information in the absence of three-dimensional sensor scenes and improve position coordinates. precision.

[0032] Based on the detection results, the posture estimation module calculates the posture angles of various parts of the athlete's body, including head pitch, torso rotation, and limb angles. In a typical approach, the body can be viewed as a model connected by 14-21 rigid joints, and key parameters such as movement amplitude and symmetry are determined by changes in joint angles.

[0033] In some embodiments, the motion posture information can be converted into tensor form and fused with subsequent instrument data. In this case, the image perception module not only provides two-dimensional target recognition capabilities, but also has the ability to construct high-dimensional posture data structures.

[0034] When processing complex actions such as hurdle jumping and aerial flips, the system will automatically detect behavioral patterns such as acceleration peaks and joint point mutations within the action range, identify action key frames, and provide reliable data support for subsequent AI classifiers.

[0035] In addition, in terms of data flow architecture, the monitoring and identification module in this embodiment has a real-time caching mechanism, which is used to cache several frames of images and feature results before and after the current frame at the local edge to support image callback and capture operations when subsequent modules make judgments.

[0036] It should be noted that the accuracy and frame rate of image data processing can be configured based on the sport. For example, in static posture judging events (such as archery), frame rate has a lower priority; while in fast-moving events (such as long jump), video data input of at least 30 FPS is required.

[0037] For the equipment tracking module, in this embodiment, based on the establishment of preliminary motion detection and posture recognition in the monitoring and recognition module, in order to further build an input structure with "human-object" spatiotemporal coupling capabilities, the equipment tracking module is introduced to achieve continuous perception and data structuring of the movement behavior of sports equipment. As an important link between image perception and AI penalty modeling, it needs to meet the collaborative requirements of high-frequency sampling, low-latency synchronization and structural pairing.

[0038] The equipment tracking module primarily serves sports involving specialized equipment, such as gymnastics, track and field, martial arts, and weightlifting. Because the athlete's movements in these events are closely related to the equipment's position, velocity, and acceleration, the module must accurately capture the equipment's dynamic parameters in real time and align them with the athlete's posture information to ensure the accuracy and consistency of the judging criteria.

[0039] In this embodiment, the device tracking module includes a tracking chip unit, a speed and acceleration measurement unit, and a data synchronization unit.

[0040] Typically, tracking chips are installed in fixed locations on competition equipment, such as the front end of the pommel horse, the trailing edge of the long jump springboard, the tail of the javelin, or the edge of the diving platform. These chips are implemented using radio frequency identification (RFID) or inertial measurement units (IMUs). In one common configuration, a micro IMU chip with nine-axis attitude resolution is used, integrating a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer.

[0041] As an option, to reduce power consumption and improve dynamic response capabilities, the tracking chip uses low-frequency Bluetooth or ZigBee to communicate with the main control. The main control can be set in the sideline processing terminal to receive all chip data and add a unified time stamp.

[0042] Specifically, the velocity and acceleration measurement unit is used to continuously collect the linear velocity and acceleration data of the device in three-dimensional space. Velocity can be obtained by position differentiation or directly through the IMU, depending on the structural characteristics of the device. Acceleration data is directly provided by the accelerometer and expressed as a three-dimensional vector: ; in, Indicates the time at which the device The acceleration vector of are the acceleration components on the three spatial axes respectively.

[0043] In some embodiments, the system can also calculate the motion path estimation trajectory based on the velocity integral: ; in, For equipment at all times speed; Indicates the device at the current moment The acceleration vector of is the sampling interval; Indicates that the device will be in the future The above trajectory estimation can provide structural boundary support for subsequent posture-device matching.

[0044] In this embodiment, the data synchronization unit is used to time-align the collected equipment position information with the athlete posture timing information obtained by the monitoring and recognition module. This alignment is achieved through a multi-stage processing strategy, including a timestamp synchronization algorithm, a data interpolation mechanism, and a modal fusion method.

[0045] Specifically, in one possible implementation, a time synchronization algorithm uses a unified time base to align the sampled data from the instrument and image devices at the frame level. If the system detects a mismatch in the sampling intervals, an interpolation algorithm is automatically activated. For example, linear or spline interpolation can be used to fill in missing data segments to achieve data continuity.

[0046] The modal fusion algorithm pairs the athlete's posture vector with the equipment's dynamic vector in frame order based on the aligned time frames, forming a structurally consistent fusion tensor structure for subsequent AI penalty model input. Each frame of data in the fusion structure contains the following: attitude angle vector; Device location coordinates; Equipment speed and acceleration; Time synchronization tag.

[0047] In some embodiments, event tags may be added to the synchronization results, such as "touch device starting point" and "instrument landing point", etc., to enhance the system's ability to identify key behavior nodes.

[0048] In this implementation, to ensure stable system operation in high-frequency motion scenarios, the device tracking module must maintain a data sampling frequency of at least 50Hz, while maintaining transmission latency within 30ms. To this end, the module incorporates a double buffering mechanism and differential compression into its data upload structure, achieving a balance between real-time data performance and transmission efficiency.

[0049] Regarding the rule parameter construction module, in this embodiment, in order to enable the AI ​​penalty model to have a structured input basis for action standards and to achieve accurate judgment of violations in different sports and multiple action scenarios, a rule parameter construction module is set up as a bridge between the core rule logic and action labels.

[0050] The traditional penalty criteria, which rely on subjective human interpretation, are transformed into structured rule representations that are system-callable and algorithm-recognizable. By encoding elements such as project rules, action requirements, and violation conditions into penalty rule vectors, the AI ​​model's ability to distinguish between "legal" and "illegal" actions is improved, while also enhancing the system's universal adaptability.

[0051] In this embodiment, the rule parameter construction module includes a rule input unit, a penalty rule vector generation unit, and a structural constraint condition generation unit.

[0052] Specifically, the rule input unit is used to receive rule configuration items uploaded by the sports event organizer or management system. Generally, the rule configuration includes multiple dimensions such as action target range, start and end time requirements, contact position, speed limit, angle limit, operation path, etc.

[0053] For example, in the long jump competition, the contact speed of the equipment in the last step before take-off must be no less than the threshold The landing point must be within the valid range The class requirements will be formatted as a vector of rule terms.

[0054] In one possible implementation, rule input uses JSON or table structure, is written into the rule database through an interface, and is periodically synchronized to edge devices to ensure on-site availability.

[0055] In this embodiment, the penalty rule vector generation unit is used to map the above rule format into a specific structure tensor. Each type of action corresponds to a set of rule vectors, and the generation method is as follows: ; in, represents a rule vector; Label for the action type (e.g., takeoff, support, swing); is the set of constraints corresponding to the action (such as speed limit, contact detection); The time window or trigger period to which the action belongs. Function Represents the rule encoding logic, typically a nested structure mapping table or a standardized matching function. The output vector will be used as the constraint input in the subsequent penalty model and compared with the perception data.

[0056] In some embodiments, the vector will also include the fuzzy interval tolerance factor , used to control the robustness of the AI ​​model to edge actions, such as allowing Errors within the range are not considered violations.

[0057] The structural constraint condition generation unit is used to combine the penalty rules with the real-time perception action data to generate penalty input caliber that can be directly called by the AI ​​algorithm. For example, the system may require that in certain high-risk actions, the action duration Cannot fall below the threshold , otherwise the action is considered incomplete: ; in, 、 They represent the start and end time nodes of the action, respectively, and are triggered by the system's automatic detection of key frames. This formula forms part of the structural time constraint and is embedded in the penalty logic diagram.

[0058] In some embodiments, to enhance the system's scalability, the rule parameter construction module also includes an "action rule sub-library" management mechanism. Rules for different disciplines are archived in a modular manner, facilitating quick switching, additions, and modifications. For example, technical disciplines like diving, gymnastics, and figure skating can each independently configure movement standards.

[0059] Normally, the default rule set is loaded when the system is initialized; if there are changes in the on-site project, it can be quickly switched through the remote port.

[0060] In actual deployment, this module is usually deployed on a local server or a central control terminal for penalty judgment, and communicates with the fusion module through a low-latency protocol to ensure that the rule logic information required for penalty judgment can be retrieved within milliseconds and participate in AI model judgment.

[0061] In this embodiment, the fusion judging module comprehensively analyzes data from the multimodal perception module, monitoring and identification module, equipment tracking module, and rule parameter construction module, and outputs the final judgment results based on the rules and technical requirements of the sports competition. By integrating multi-source information, the accuracy and fairness of the judgment are ensured.

[0062] The fusion judgment module adopts a weighted fusion strategy to weight the information from different modules and realize comprehensive analysis and intelligent judgment of multi-dimensional data through the weighted results.

[0063] Specifically, the output results of the fusion penalty module include: the legality judgment of the athlete's behavior, the detection of illegal behaviors during the execution of the action, and the final penalty decision information.

[0064] In its implementation, the core of this module is to generate penalty decisions by calculating and fusing feature vectors and applying a rule-based model. This process uses various sensory inputs, including the athlete's real-time motion state, equipment motion data, movement characteristics, and rule parameters. This data is then applied through a weighted function to form a comprehensive evaluation index, which determines whether the movement is legal.

[0065] In general, the calculation process of the fusion penalty module includes the following steps: Data fusion: In this stage, the data from the monitoring and identification module, the instrument tracking module, and the rule parameter construction module are integrated. Each type of data has a specific weight coefficient, which is used to represent the contribution of the data to the final penalty decision. Assume that the penalty feature vector in the fusion process is expressed as , which consists of action monitoring data and device tracking data By weighted combination we get: ; in, and is a weighting coefficient, which indicates the importance of monitoring data and device tracking data in the final penalty decision; Feature data extracted for monitoring recognition module; It is the characteristic data of the device tracking module.

[0066] Compliance Assessment: During the compliance assessment phase, the fused data is analyzed to determine whether the athlete's movements comply with the rules. These assessments are typically based on factors such as the athlete's movement type, movement path, and movement duration. The system performs compliance analysis based on the rule parameters, for example, calculating whether the athlete's movements meet the start, duration, and end time limits, as well as speed and angle requirements.

[0067] For each specific action, the system generates a compliance score , which is expressed as: ; in; For the The probability value of the class action; The reliability of the final penalty result; The total number of action categories. This formula determines the final penalty result by selecting the maximum compliance score. This score represents the system's evaluation of each action, with the highest score determining whether the action is compliant.

[0068] Output of penalty results: After the compliance determination is completed, the fusion penalty module generates the final penalty decision results based on the compliance score. These decision results include whether there is a violation, the type of violation, and the specific judgment of the action execution. The reliability value of the final penalty result is calculated based on the weighted calculation of the rule model to indicate the reliability of the penalty result. Specifically, the reliability calculation formula is: ; in, Indicates the confidence level of the final penalty result; is the compliance score; It is the time factor in the penalty process; and is the weighting coefficient, which represents the impact of different penalty factors on the final reliability value.

[0069] In one possible implementation, the fusion officiating module can also dynamically adjust based on real-time data, continuously optimizing weighting coefficients through learning algorithms (e.g., deep learning networks). As data accumulates, the system can automatically adjust weights based on historical officiating data and individual player behavior patterns, thereby improving the accuracy and fairness of officiating.

[0070] Regarding the output and capture module, in this embodiment, after the fusion judgment module completes the dynamic comparison of multimodal perception results with structured rule constraints, it must simultaneously trigger the capture and backtracking of on-site images and the output of judgment labels to achieve a visual presentation of action violations. To this end, the system has set up an output and capture module, which is responsible for the image-driven conversion of judgment results, time-series capture control, and the synchronous return of image and label data. It is an important bridge connecting the back-end AI judgment logic with the user's front-end visual display.

[0071] The output and capture module includes three submodules: image driver unit, image acquisition trigger unit, and image upload and presentation unit. Its functions and technical implementation are as follows: The image driver unit receives the penalty result data output by the fusion penalty module and triggers the image capture action based on the corresponding penalty tag. To improve the accuracy of the penalty response, this unit uses a reverse window prediction mechanism to infer the image frame interval to be captured from the action start and end time tags received by the fusion module.

[0072] Specifically, the image capture period is determined by the following formula: ; in, Indicates the time interval of the image to be captured; and are the start and end times of action recognition respectively; is the duration of the preceding guide frame; The duration of the post-end frame.

[0073] This time window is paired with the frame sequence in the video buffer through a soft synchronization strategy to achieve high consistency backtracking of illegal action penalties and visual evidence.

[0074] In this embodiment, the image acquisition trigger unit is implemented through a hardware-software collaborative trigger mechanism. The hardware communicates with the camera controller, and the software searches for the corresponding frame based on a timestamp-to-frame number mapping table. The acquisition signal is provided by the image driver unit, and the specific frame number is located using the following formula: ; in, To capture the The sequence number corresponding to the frame; Indicates the start time of the video sequence; Indicates the interval between video frames (such as 40ms); To capture the first The time point of the target frame.

[0075] As an option, the module can be set to asynchronously capture multiple frame intervals and perform sliding comparisons to select the frame group with the best clarity and target occlusion for output, thereby improving the quality of image evidence.

[0076] In one possible implementation, to reduce bandwidth load, the module supports on-demand compression transmission. Common strategies include: frame selection and cropping, resolution degradation, tag-first transmission and other modes to adapt to terminal reading requirements in different network environments.

[0077] Typically, this module is deployed on edge servers and features local caching and a mechanism for resuming transmission after an exception. If a camera drops frames or the video stream is interrupted, the system automatically switches to a backup path or recalibrates the frame registration logic to ensure that captured information is not lost and the penalty visualization link remains uninterrupted.

[0078] Regarding the wireless linkage and energy management module, in this embodiment, while the system generates the penalty results for sports behaviors and the output module completes image annotation and transmission, it still needs to quickly and efficiently synchronize the system penalty information to terminal devices (such as signal lights, buzzers, and linesman tablets) to achieve physical response, status display, or auxiliary judgment instructions at the event site. To this end, the system has a wireless linkage and energy management module specifically designed to manage remote wake-up control and working state switching of terminal devices, simultaneously fulfilling the dual functions of communication scheduling and energy supply efficiency control.

[0079] The wireless linkage and energy management module includes a linkage instruction generation unit, a wireless communication unit, and an energy status monitoring unit. The specific functions and working process are as follows: The linkage command generation unit is responsible for generating terminal response commands based on the structured penalty results output by the fusion penalty module or the event tag field in the image tag. This unit uses an event-device mapping table structure to bind different penalty action categories to control commands and assigns priority tags to distinguish mandatory responses from recommended responses.

[0080] In one possible implementation, the instruction generation logic is encoded based on the following mapping function: ; in, Indicates the Linkage control instructions; The action type label of the current penalty event; Number the linkage equipment in the venue; The penalty type, such as "Line violation" and "Two-handed assistance"; Represents the rule matching and priority fusion function, which is generally a hash table index function or a state jump matrix.

[0081] The wireless communication unit is used to send the above control instructions to the corresponding device. This system supports multiple short-range wireless protocols, including but not limited to BLE 5.0, Wi-Fi 6, ZigBee, etc., and can adaptively switch based on the communication interference conditions at the site.

[0082] To improve communication stability, the module uses a time-slice broadcast + device perception response mechanism to ensure that the key control signal reachability exceeds the threshold within each task scheduling cycle. The communication time budget can be estimated using the following formula: ; in, Indicates the total time taken for a complete frame of instructions from generation to confirmation; is the packet length (bit); is the effective bandwidth (bps); Processing overhead for the terminal (e.g., unpacking, inspection); To confirm feedback delay.

[0083] Generally, this module sets the scheduling cycle to 30-50ms, which can still ensure that the linked devices respond to tasks within 100ms in large-scale event scenarios.

[0084] The Energy State Monitoring Unit (ESMU) detects the device's operating power consumption and idle state, establishing a multi-level energy state machine ("operating-standby-sleep"). Optionally, the device can automatically enter a low-power standby mode during idle windows based on task intensity and pre-set load policies.

[0085] In some embodiments, the terminal device determines whether to enter sleep mode by detecting the communication heartbeat packet loss rate or actively feeding back the duration of the no-task state.

[0086] Once the system issues a new penalty task, the energy module sends a wake-up signal, quickly pulls the device out of the low-power state through hardware interruption or power boost, and enters the response preparation.

[0087] The real-time sports competition penalty judging method based on multimodal sensing and AI algorithm described below and the real-time sports competition penalty judging system based on multimodal sensing and AI algorithm described above can be used in correspondence with each other.

[0088] Please see the attached Figure 2 The present invention also provides a real-time sports competition penalty judgment method using multimodal sensing and AI algorithm, comprising the following steps: The monitoring equipment collects image data including athlete images and postures, and the tracking chip collects equipment data. The image data and equipment data are processed synchronously to build a multimodal perception input structure that includes posture timing and equipment dynamics. Based on the action categories, motion trajectories, and equipment features in the multimodal perception input structure, the system calls the preset rule library for sports events, generates penalty rule vectors related to the detected actions, and dynamically loads structural constraints. Based on the multimodal input structure and penalty rule vector, the image features and device features are aligned and weighted fused to generate a fused feature tensor; The fused feature tensor is input into the trained AI penalty model to perform action compliance classification calculations and generate penalty output results including action category labels, violation determination results, and confidence values; Drive the image capture logic based on the penalty results, extract the video frame at the corresponding time point, generate the penalty image record, and synchronize the penalty image record and penalty label to the on-site electronic display screen and user terminal; Based on the task load and operating status of the penalty system, the power level of each device is monitored in real time, and the power consumption adjustment strategy is set according to the task intensity.

[0089] The method of this embodiment can be used to execute the above system embodiment. Its principles and technical effects are similar and will not be described in detail here.

[0090] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A real-time sports competition penalty judging system based on multimodal sensing and AI algorithms, characterized by: include: The monitoring recognition module is used to perform human figure recognition and motion tracking based on image data collected by monitoring equipment, generating target detection results and posture timing information; The equipment tracking module includes a tracking chip installed on the sports equipment, which is used to collect the equipment's position information, velocity data, and acceleration parameters through the tracking chip, and synchronize the equipment motion data with the posture timing information to build a multimodal perception input structure; A rule parameter construction module is used to receive the preset sports game rule configuration, generate a penalty rule vector that matches the action category and violation conditions, and provide structural constraints for AI penalty decision-making; The fusion penalty module is used to construct a fusion feature tensor based on the multimodal perception input structure and the penalty rule vector, and input it into the AI ​​model for action compliance classification to generate penalty results and confidence information; The output and capture module is used to drive the image capture logic according to the penalty results, generate penalty image records of the corresponding frames, and synchronize the penalty results and image data to the electronic display screen and user terminal; The wireless linkage and energy management module is used to establish a low-latency wireless communication link, monitor the power status of the real-time penalty system, dynamically adjust the power consumption strategy according to the perceived task intensity, manage the device charging status and control the system linkage.

2. The multimodal sensing and AI algorithm sports competition real-time penalty judging system according to claim 1 is characterized in that: The monitoring and identification module includes: The target detection unit is used to detect human targets in image data, extract the athlete's contour information and action key points, and perform target positioning and recognition based on convolutional neural networks; The motion tracking unit is used to track the athlete's motion changes through a time-series tracking algorithm, generate motion trajectories and posture change time series, and establish the athlete's real-time motion model; The motion analysis unit is used to classify and analyze the detected movements, generate specific labels for each movement based on historical data, and identify whether the movement complies with the competition rules.

3. The real-time sports competition penalty judging system based on multimodal sensing and AI algorithm according to claim 2 is characterized in that: The performing human target detection on image data comprises: Athlete position coordinates, representing the athlete's position in the competition venue, are calculated as follows: ； in, Indicates the current position of the athlete. are the image coordinates of the athlete, is the acceleration data; Action key points, using deep learning methods to identify the key points of athletes and map them to the field coordinate system; The posture estimation results estimate the posture of each part of the athlete's body based on the athlete's image data, including the angle information of the torso and limbs.

4. The multimodal sensing and AI algorithm sports competition real-time penalty judging system according to claim 1 is characterized in that: The device tracking module includes: Tracking chip units are installed on various devices and use radio frequency identification technology to collect the device's location information and movement status in real time; The speed and acceleration measurement unit is used to record the movement speed and acceleration data of the instrument, and capture the acceleration changes of the instrument at different time points through sensors; The data synchronization unit is used to synchronize the position information of the equipment with the athlete's posture timing information.

5. The real-time sports competition penalty judging system based on multimodal sensing and AI algorithm according to claim 1 is characterized in that: The rule parameter building module includes: A rule input unit for receiving action determination rules configured by the competition organizer, including specific action standards and violation conditions for different sports; A penalty rule vector generation unit generates a penalty rule vector that complies with the competition specifications based on the received rule information and stores it in a data structured manner; The structural constraint generation unit combines real-time action data with competition rules to generate constraints that need to be followed during the penalty process, including action legality, time window and speed limit.

6. The multimodal sensing and AI algorithm sports competition real-time penalty judging system according to claim 1 is characterized in that: The fusion penalty module includes: The feature fusion unit is used to merge the data from the monitoring and recognition module and the device tracking module, and generate a feature fusion tensor through the weighted average method: ； in, Indicates the comprehensive penalty result. To monitor data characteristics, is the device data feature, and is the weight; The action compliance classification unit is used to judge the compliance of the action based on the fused feature tensor and output the legality probability and penalty level of the action; The confidence calculation unit is used to calculate the confidence of the penalty result and output the penalty result and its credibility index.

7. The real-time sports competition penalty judging system based on multimodal sensing and AI algorithm according to claim 1 is characterized in that: The output and capture module includes: An image driving unit, configured to generate a corresponding image capture instruction according to the penalty result output by the fusion penalty module; An image acquisition trigger unit, configured to control the monitoring camera device to capture an image at a specific time point based on the capture instruction; The image upload and presentation unit is used to synchronously transmit the captured image data and penalty label information to the terminal presentation screen or the user's visual terminal.

8. The multimodal sensing and AI algorithm sports competition real-time penalty judging system according to claim 1 is characterized in that: The wireless linkage and energy management module includes: A linkage instruction generation unit, configured to generate a trigger instruction for controlling a linkage response of a terminal device based on a system penalty result or sensory feedback; a wireless communication unit, configured to send the trigger instruction to a target device via Bluetooth, Wi-Fi, or other short-range wireless communication protocols; The energy status monitoring unit is used to monitor and adjust the energy supply status of the terminal device in real time, automatically enter low-power mode when the device is in standby mode, and quickly wake up to respond to control when receiving linkage tasks.

9. The multimodal sensing and AI algorithm sports competition real-time penalty judging system according to claim 6 is characterized in that: The penalty result is calculated by the confidence function: ； in, For the The probability value of the class action, is the reliability of the final penalty result, The total number of action categories.

10. A real-time sports competition penalty judging method using multimodal sensing and AI algorithms, applied to a real-time sports competition penalty judging system using multimodal sensing and AI algorithms as described in any one of claims 1 to 9, characterized in that: The following steps are involved: The monitoring equipment collects image data including athlete images and postures, and the tracking chip collects equipment data. The image data and equipment data are processed synchronously to build a multimodal perception input structure that includes posture timing and equipment dynamics. Based on the action categories, motion trajectories, and equipment features in the multimodal perception input structure, the system calls the preset rule library for sports events, generates penalty rule vectors related to the detected actions, and dynamically loads structural constraints. Based on the multimodal input structure and penalty rule vector, the image features and device features are aligned and weighted fused to generate a fused feature tensor; The fused feature tensor is input into the trained AI penalty model to perform action compliance classification calculations and generate penalty output results including action category labels, violation determination results, and confidence values; Drive the image capture logic based on the penalty results, extract the video frame at the corresponding time point, generate the penalty image record, and synchronize the penalty image record and penalty label to the on-site electronic display screen and user terminal; Based on the task load and operating status of the penalty system, the power level of each device is monitored in real time, and the power consumption adjustment strategy is set according to the task intensity.

Images