Multi-modal perception interactive targeting game judgment feedback method

Abstract

The invention discloses a multi-modal perception interactive shooting game determination feedback method, and relates to the technical field of man-machine interaction and multi-modal perception, and the method comprises the following steps: collecting a lamp driving frequency and a camera shutter frequency, determining a frequency intersection region of the lamp driving frequency and the camera shutter frequency, generating a light field interference pattern, the interference relation between the illumination rhythm and the image sampling rhythm is presented in a visual mode; a highlight burst trajectory is extracted based on the light field interferogram, and a transient reflection seed set is constructed and used for defining the spatial and temporal distribution range of abnormal light spots caused by illumination stroboflash. Through double-end dynamic regulation and control of phase respiration slippage of a lamp driving end and microstep wandering of a camera shutter end, a flash avoidance corridor is formed on a time axis, so that an illumination peak value and an exposure window keep avoiding, stroboscopic interference is eliminated, a false impact point is extinguished, and stable picture brightness and consistent recognition rhythm are ensured; therefore, the judgment precision and the feedback stability of the interactive target shooting game are improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and multimodal perception technology, specifically to a multimodal perception-based interactive shooting game judgment and feedback method. Background Technology

[0002] Multimodal perception-based interactive shooting game judgment and feedback refers to the technical process in interactive shooting games that intelligently judges the player's shooting behavior and generates real-time feedback by integrating multiple sensory information such as vision, hearing, motion, and touch. The system uses a camera to recognize the player's posture and shooting trajectory, combined with multimodal data such as voice commands collected by the microphone and vibration or contact signals detected by sensors. Through algorithm analysis, it accurately judges indicators such as target hit, hit location, and reaction speed, and triggers multi-dimensional responses such as light effects, sound effects, or force feedback based on the judgment results, giving the game an immersive interactive experience and adaptive judgment capabilities.

[0003] The existing technology has the following shortcomings: In existing technologies, the visual judgment stage of interactive shooting games typically relies on camera image acquisition results for bullet impact point identification, but it does not fully consider the dynamic changes in scene lighting. When the lighting device uses a high-frequency flickering light source, if its flicker period interferes with the camera shutter sampling frequency, transient reflection points will appear in the image frame. These transient reflection points last only a very short time dimension, but exhibit high brightness characteristics similar to real bullet impact points in spatial brightness distribution, making it difficult for visual recognition algorithms to distinguish between real hit signals and optical interference pseudo-signals under current technological conditions. As a result, the recognition result may misclassify the reflection point as a valid bullet impact point, thereby triggering false hit feedback, causing judgment result deviation, feedback rhythm disorder, and scoring logic confusion, seriously weakening the accurate judgment capability and feedback stability of interactive shooting games.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal perception-based interactive shooting game judgment feedback method to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal perception-based interactive shooting game judgment and feedback method, comprising the following steps: The driving frequency of the lighting fixtures and the shutter frequency of the camera are collected, the frequency intersection area of ​​the two is determined, and a light field interferogram is generated to visualize the interference relationship between the illumination rhythm and the image sampling rhythm. High-brightness burst trajectories are extracted based on light field interferograms, and a set of transient reflection seeds is constructed to define the spatiotemporal distribution range of abnormal light spots caused by illumination flicker. A time anchor sequence is established around the transient reflection seed set, phase misalignment frames are extracted, and a misalignment frame list is generated to describe the temporal distribution characteristics of illumination flicker. Based on the list of misaligned frames, a reflection isolation window is set up to adjust the image retrieval order during the bullet impact point recognition process and output a cleaned image data stream to actively isolate optical reflection interference in the visual recognition path. Dual-end dynamic control is performed in the purified image data stream. By synchronously controlling the phase breathing slide of the lamp driver end and the micro-step movement of the camera shutter end, a flash avoidance corridor is formed, so that false impact points are actively extinguished and the hit recognition rhythm is kept stable.

[0007] Preferably, the process for generating the light field interferogram is as follows: The light emission control signal from the driver end of the lighting device and the exposure trigger signal from the shutter end of the image acquisition device are collected, and two sets of time series data are recorded under a unified time reference, so that the waveform of the change in lighting brightness and the exposure opening curve are kept synchronized on the time axis. Based on the synchronized time series, determine the intersection zone of the illumination brightness change cycle and the exposure time window on the time axis, record the start time, end time and overlap duration of each intersection zone, and form the coupling interval between the illumination rhythm and the sampling rhythm. The illumination intensity time curve and the exposure curve are mapped onto the spatial plane according to the time correspondence to generate a light field interference distribution structure containing alternating bright and dark fringes, which is used to reflect the phase relationship between the two. The brightness of the light field interference distribution structure is expanded in time and extended in space to form a three-dimensional light field interference spectrum, so that the interference relationship between the illumination rhythm and the sampling rhythm can be presented in a visual way.

[0008] Preferably, the steps for constructing the transient reflection seed set are as follows: After the light field interferogram is generated, the brightness change value is read line by line along the time axis, the time periods of brightness increase and decrease are recorded, the high brightness burst segments are identified and their spatial coverage is marked. Extend and connect consecutive or overlapping bright burst segments in spatial coordinates within adjacent time periods to form a set of bright burst trajectories spanning multiple sampling periods; The instant with the fastest increase in brightness and the highest peak value is extracted along the direction of the high brightness burst trajectory as the transient reflection candidate point, and the time position, spatial position, brightness peak value and duration are recorded to form a transient reflection seed set; Transient reflection seeds are merged continuously in the time dimension and aggregated according to proximity in the spatial dimension to generate an abnormal light spot distribution map, which is used to define the range of reflection interference caused by lighting flicker.

[0009] Preferably, when generating the abnormal light spot distribution map, the center position and boundary contour of the reflection area unit are determined by comprehensively weighting the brightness peak, duration and spatial diffusion range of the transient reflection seed, so that the time span and spatial concentration of each abnormal light spot in the reflection distribution map are consistent, thereby improving the accuracy and stability of the spatiotemporal distribution definition of abnormal light spots.

[0010] Preferably, the steps for generating the misaligned frame list are as follows: After the transient reflection seed set is established, the time coordinates of all seeds are sorted and a continuous time node chain is formed. Based on the driving cycle of the lighting device, the node with the largest brightness peak is extracted to establish a time anchor sequence. Rising anchors, peak anchors, and falling anchors are set between adjacent time anchors to fully record the changing trend of light intensity within one cycle, forming a continuous time anchor sequence; Extract the corresponding image frame interval with the time anchor point as the center, analyze the time position of the brightness peak, identify positive phase misalignment and negative phase misalignment, and form a set of phase misalignment frames; The phase misalignment frames are archived in the order of sampling time, and the misalignment direction, duration and brightness difference are recorded to generate a misalignment frame list, which is used to describe the temporal distribution characteristics of light flicker.

[0011] Preferably, the distribution of each time anchor point in the time anchor point sequence is determined according to the rate of change of illumination brightness. The rising anchor point corresponds to the point where the brightness growth rate is the highest, the peak anchor point corresponds to the turning point where the brightness change changes from increasing to decreasing, and the falling anchor point corresponds to the point where the brightness decay rate is the largest, so that the intensity change process within the illumination cycle can be completely marked on the time axis.

[0012] Preferably, the steps for outputting the purified image data stream are as follows: Based on the temporal distribution information of the misaligned frame list, the temporal range of the lighting interference is determined, a reflection isolation window with front and back boundaries is established, and a buffer time is set on the boundary to form a temporal isolation framework. The reflection isolation window is mapped onto the image frame sequence. By comparing the timestamp with the spatial coordinate data in the transient reflection seed set, the spatial coverage of the illumination reflection is determined, forming a reflection isolation window structure with spatiotemporal constraints. During the bullet impact point recognition process, the image frame retrieval order is adjusted according to the reflection isolation window. Interference frames that fall into the window in both time and space are removed and transition frame information is inserted to maintain the continuous recognition rhythm. The updated recognition frame sequence is time-smoothed and brightness-compensated for recombination, and a cleaned image data stream is output to achieve active isolation of optical reflection interference.

[0013] Preferably, the time boundary of the reflection isolation window is adaptively adjusted according to the rate of brightness change. When the rate of brightness increase is higher than the average rate of exposure change, the front boundary extends forward, and when the rate of brightness decay is lower than the normal rate of decrease, the rear boundary extends backward, so as to ensure that the time range of reflection interference is fully covered and to maintain the temporal continuity of the purified image data stream.

[0014] Preferably, the dual-end dynamic control is performed in the purified image data stream. By synchronously controlling the phase breathing slide of the lamp driver end and the micro-step movement of the camera shutter end, the following steps are taken to form a flash avoidance corridor on the time axis: After the purified image data stream output is stabilized, the light emission control signal of the lamp driver end and the exposure trigger signal of the camera shutter end are synchronously acquired to record the phase offset between the peak time of illumination and the center time of the exposure window, and to establish a time reference for dual-end dynamic control. The timing of the start of the light emission cycle at the driver end of the lamp is controlled by the phase shift direction and amplitude, so that the light peak moves periodically on the time axis, forming a phase breathing shift. Using the real-time phase information of the lamp's light emission signal as a reference, the sampling window is exposed within the low-brightness range of the illumination cycle by adjusting the shutter opening time or exposure duration. Synchronize and align the timing information of the lamp phase breathing slide with the shutter microstep movement to form a flash avoidance corridor between the falling segment of the illumination waveform and the rising segment of the sampling window, thereby achieving active extinguishing of false impact points and maintaining a stable hit recognition rhythm.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention establishes a dual-end dynamic control mechanism between the illumination driving end and the image acquisition end, creating a continuously staggered anti-flash corridor between the illumination emission cycle and the camera sampling cycle on the time axis, fundamentally eliminating transient reflection interference under high-frequency flash illumination conditions. Dynamic separation of the illumination peak and exposure window is achieved through phase breathing slip at the lamp driving end and micro-stepping at the camera shutter end. False impact points are actively extinguished during imaging, restoring stable image brightness distribution and ensuring the purity and continuity of the target shooting image data stream during the sampling phase, providing a reliable visual input basis for the judgment process.

[0016] This invention achieves stable and consistent target recognition rhythm by continuously implementing dual-end dynamic control within the purified image data stream, automatically maintaining phase balance between the illumination rhythm and the sampling rhythm during operation. As the lightning avoidance corridor forms and persists, image brightness fluctuations are suppressed, the triggering sequence of the recognition signal remains constant, and the rhythmic stability and response accuracy of the judgment feedback process are simultaneously improved. Through this control method, interactive target shooting games can still achieve a smooth recognition experience and consistent interactive feedback even in complex lighting environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of the multimodal perception interactive shooting game judgment feedback method of the present invention. Detailed Implementation

[0019] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0020] This invention provides, for example Figure 1 The multimodal perception-based interactive shooting game judgment feedback method shown includes the following steps: The driving frequency of the lighting fixtures and the shutter frequency of the camera are collected, the frequency intersection area of ​​the two is determined, and a light field interferogram is generated to visualize the interference relationship between the illumination rhythm and the image sampling rhythm. To visualize the interference relationship between illumination rhythm and image sampling rhythm, a light field interferogram is constructed by precisely capturing, synchronizing, spatially mapping, and dynamically unfolding the temporal characteristics of the illumination end and the image acquisition end. This process includes the following steps: Frequency parameters are acquired at the drive end of the lighting device and the shutter end of the image acquisition device, respectively. The drive section of the lighting device outputs a current signal controlled by fixed pulse width modulation, which directly determines the brightness variation rhythm of the light source. By setting high-precision time sampling points in the drive circuit, the rising edge, steady-state segment, and falling edge of the current waveform are recorded to obtain the duration, peak duration ratio, and luminous flux change rate of each illumination cycle. Simultaneously, on the image acquisition device side, the periodic trigger signals for shutter opening and closing are acquired through the exposure control circuit, recording the start time of each exposure opening, the exposure duration, and the closing time. To ensure that the time references of the lighting and acquisition ends are consistent, this step synchronizes the two sampling channels using a unified clock reference signal, so that the lighting signal and the shutter signal are in the same reference frame on the time axis. After synchronized acquisition, two sets of continuous time series data are obtained: one set is the drive waveform of brightness variation over time, and the other set is the sampling rhythm curve of exposure opening and closing, laying the time foundation for the subsequent determination of the frequency intersection zone.

[0021] After obtaining the illumination drive waveform and shutter trigger curve, the frequency intersection region between them on the time axis is determined. The frequency intersection region refers to the portion where the periodic curve of illumination intensity change overlaps with the image exposure time window in time. In this process, using the period of the illumination waveform as a reference, the start and end times of exposure at the image acquisition end are mapped onto the illumination waveform in chronological order. For each exposure cycle, its overlapping interval with the illumination waveform is identified, and the start time, end time, and overlap duration of this interval are recorded. Subsequently, the overlap distribution of multiple consecutive cycles is compared sequentially to define the area matching the exposure window in each illumination cycle. Through this continuous comparison, the intersection distribution sequence between the illumination brightness cycle and the exposure cycle is obtained. This distribution sequence exhibits a periodic variation pattern; when the illumination frequency and sampling frequency are close, the intersection region slides on the time axis, resulting in frame-by-frame misalignment. The center position of each intersection region represents the synchronous superposition point of a peak illumination brightness and a highly sensitive exposure period. The collection of all intersection regions constitutes the coupling interval between the illumination rhythm and the sampling rhythm, providing spatial positioning basis for generating the light field interferogram.

[0022] After determining the frequency intersection region, the time-varying curve of illumination intensity and the exposure curve of image sampling are mapped onto a spatial plane according to their time correspondence, generating the basic distribution structure of the light field interferogram. Specifically, the illumination periodic waveform is continuously plotted on the plane with the time axis as the horizontal coordinate and the illumination intensity change value as the vertical coordinate. On the same plane, the exposure window of the image acquisition end is superimposed in the form of rectangular strips, where the length of the strip represents the exposure duration and the position of the strip corresponds to the time period during which the exposure occurs. When the brightness peak region of the illumination waveform overlaps with the exposure strip, a brighter crossband is formed in that region on the plane. When the overlapping results of multiple consecutive frames are superimposed, a periodically distributed structure of alternating bright and dark fringes is formed on the entire plane. The density, width, and tilt angle of these fringes reflect the phase relationship between the illumination flicker period and the shutter sampling period. When the illumination frequency is slightly higher than the sampling frequency, the fringes tend to shift towards the positive time axis, exhibiting interference drift characteristics; when the illumination frequency is slightly lower, the fringes are in the opposite direction, exhibiting interference backlash characteristics. This planar fringe distribution pattern constitutes the preliminary form of the light field interferogram, which intuitively demonstrates the coupling mode between illumination changes and sampling rhythm.

[0023] After obtaining the planar structure of the light field interferogram, its brightness distribution is temporally unfolded and spatially extended to form a complete visual image of the light field interferometry. Specifically, this involves stacking multiple consecutive frames of light field interferograms along the time axis, accumulating the brightness information from each frame in the vertical direction to form a three-dimensional light field structure with depth information. In this structure, the peak regions of the illumination period form continuous high-brightness channels, the shutter exposure time window forms periodic bright band channels, and the intersection of these two types of channels presents a strong brightness convergence zone. The density changes in these convergence zones reflect the changing trend of the coupling strength between the illumination luminous flux and the sampling time window. When there is a slight phase difference between the light source flicker waveform and the shutter sampling rhythm, the brightness convergence zone slowly slides along the time axis, forming a tilted light band; when the frequencies of the two are perfectly synchronized, the light band is vertical, and the brightness distribution is most concentrated. Through this three-dimensional unfolding method, the interference relationship between the illumination rhythm and the image sampling rhythm is concretized into a directly observable light field interferogram. In this atlas, the changing trajectory of brightness distribution clearly reflects the phase shift between illumination and sampling, providing an intuitive basis for identifying optical interference sources. Simultaneously, the light field interferogram can also be used to assess the impact of illumination changes on imaging stability, providing a spatial reference for subsequent extraction of high-brightness burst trajectories and transient reflection localization.

[0024] High-brightness burst trajectories are extracted based on light field interferograms, and a set of transient reflection seeds is constructed to define the spatiotemporal distribution range of abnormal light spots caused by illumination flicker. To accurately identify abnormal light spots caused by illumination flicker and clarify their temporal and spatial distribution, it is necessary to extract high-brightness burst trajectories based on the light field interferogram and construct a transient reflection seed set accordingly. The specific implementation steps are as follows, assuming full utilization of the illumination and sampling interference relationship reflected in the light field interferogram: After the light field interferogram is generated, its brightness information is analyzed in layers to extract high-brightness burst regions that appear during illumination changes. Each pixel in the light field interferogram corresponds to the interaction between illumination brightness and image sampling time. Sudden increases in brightness typically represent the instant when the illumination peak and the shutter exposure window completely overlap in time. To identify these high-brightness burst regions, brightness values ​​are read line by line along the time axis of the light field interferogram, and the magnitude of brightness change at each moment is recorded. When the magnitude of brightness change exceeds the upper limit of the average change range of the illumination period, that time point is marked as the start of the brightness burst; continuing to read, when the brightness change returns to a stable level, it is marked as the end of the brightness burst. Thus, each continuous interval of brightness increase and decrease constitutes a high-brightness burst segment. Spatially, the horizontal coordinates of the light field interferogram are scanned column by column to determine the spatial coverage of each brightness burst segment, including the spatial location of the illumination peak, the edge coordinates of brightness diffusion, and the transition interval of brightness decrease. By scanning in both time and space, multiple discrete bursts of high brightness can be obtained, each representing an instantaneous brightness enhancement phenomenon caused by illumination interference.

[0025] After obtaining the bright burst segments distributed in the light field interferogram, these segments need to be extended and connected in both time and space dimensions to form a continuous bright burst trajectory. Each bright burst segment has characteristic information such as start point, end point, spatial location, and duration. In this step, burst segments within adjacent time periods are compared according to the continuity of their spatial locations. When two segments are adjacent in time and partially overlap in spatial coordinates, they are connected into the same bright trajectory. Through continuous connection, a brightness trajectory line spanning multiple sampling periods is formed. For segments that do not completely overlap but whose spatial distance is less than a preset threshold, they are extended and connected in chronological order, so that the trajectory line shows a smooth temporal change trend in the light field interferogram. Each formed bright trajectory line appears as a brightness band extending along the time axis in the light field interferogram, with its brightness value changing periodically over time. This extended trajectory not only reflects the periodic influence of illumination flicker on imaging brightness but also reflects the dynamic phase relationship between the illumination peak and the sampling time sequence. By using this trajectory extension method that combines temporal continuity and spatial correlation, it is possible to transform originally discrete bursts of brightness into a set of high-brightness burst trajectories with coherent temporal clues in the light field interferogram.

[0026] After forming a set of high-brightness burst trajectories, the local peak features of each trajectory are extracted to construct a transient reflection seed set. Each trajectory contains multiple brightness change peaks and troughs, where the part with the fastest brightness increase, highest peak, and shortest duration usually corresponds to a transient reflection point caused by illumination flicker interference. In this step, the brightness change curve is analyzed along the time direction of each trajectory. When the brightness value increases sharply in a very short time and quickly falls back to the reference brightness, the peak position is determined as a transient reflection candidate point. For each candidate point, its time coordinates, spatial location, peak brightness value, and duration of brightness change are recorded. Subsequently, a fixed time window is extended forward and backward from the candidate point to determine the time range of the reflection event, and a square area of ​​the same distance is extended outward from the candidate point to determine the spatial influence range of the reflection event. All candidate points that meet the criteria of short duration, high peak brightness, and limited spatial diffusion range are included in the transient reflection seed set. At this point, each seed in the transient reflection seed set corresponds to a specific reflection event, containing complete information such as time location, spatial location, brightness intensity, and duration, forming the basic unit of the reflection event.

[0027] After forming a set of transient reflection seeds, the seeds within the set are aggregated according to temporal continuity and spatial proximity to define the spatiotemporal distribution range of abnormal light spots caused by illumination flicker. This step first merges seeds with time intervals less than one-third of the illumination cycle in the temporal dimension, forming temporally continuous reflection event groups. Each group of seeds represents multiple reflections generated during the same illumination flicker peak. Then, in the spatial dimension, seeds belonging to the same time group and with a spatial distance less than a predetermined pixel threshold are spatially merged to form reflection region units. Each reflection region unit includes its center coordinates, time span, brightness concentration, and region boundary range. After all time groups and spatial units are formed, the reflection distribution of the entire light field interferogram is integrated to generate an abnormal light spot distribution map. This distribution map, with time as the vertical axis and spatial location as the horizontal axis, visually displays the occurrence time and spatial range of each abnormal reflection region in the time sequence. In this way, the periodicity of reflection interference caused by illumination flicker in time and its concentrated distribution in space can be clearly identified, providing a clear basis for the subsequent establishment of reflection isolation windows. Thus, the entire process from extracting the burst trajectory of high brightness to constructing the set of transient reflection seeds and defining the spatiotemporal range of abnormal light spots is completed, resulting in a visualization of the effects of illumination interference.

[0028] A time anchor sequence is established around the transient reflection seed set, phase misalignment frames are extracted, and a misalignment frame list is generated to describe the temporal distribution characteristics of illumination flicker. To fully reveal the temporal misalignment between illumination flicker and image sampling, a time anchor sequence needs to be established based on the existing transient reflection seed set. The placement of these time anchors clarifies the rhythmic distribution of illumination changes. This sequence is then used as a time reference to extract phase-misaligned frames, and a list of misaligned frames is generated, thus presenting the temporal distribution characteristics of illumination flicker in a structured form. The specific steps are as follows: After establishing the transient reflection seed set, the temporal information within it is systematically sorted and ordered to obtain the overall distribution of illumination change events along the temporal dimension. The transient reflection seed set records the occurrence time, spatial location, peak brightness, and duration of each sudden illumination event; this data forms the basis for constructing the time anchor sequence. In this process, the temporal coordinates of all seeds are first extracted and arranged chronologically to form a continuous time chain from the initial sampling time to the last sampling time. To avoid time overlap or omissions, the time intervals of adjacent seeds are compared during sorting. When the interval is less than half the camera exposure cycle, these seeds are considered composite events within the same time period, and only the seed with the highest brightness value is retained as the representative. When the interval is greater than twice the exposure cycle, an intermediate time point is inserted into the time gap as a supplementary marker to maintain the continuity of the time sequence. In this way, a complete time node chain is obtained, with each node representing a critical moment of a reflection event caused by an illumination fluctuation. Subsequently, using the actual driving cycle of the lighting device as the time reference, the time node chain is divided into multiple time segments, each corresponding to a complete cycle of illumination flicker. In each segment, the time node with the largest brightness peak is selected as the main anchor point of that period, thus forming the initial framework of the time anchor point sequence.

[0029] After the initial time anchor sequence is formed, to ensure that the sequence accurately describes the changing trend of light intensity, the distribution of time anchor points needs to be further refined and supplemented to establish a complete time anchor sequence. The actual shape of the lighting flicker waveform often includes three stages: rising segment, peak segment, and falling segment. Relying solely on the main anchor point cannot accurately reflect the light changes in these three stages. Therefore, this step sets multiple auxiliary anchor points between each adjacent main anchor point to capture the gradual change in light intensity. Specifically, several sub-points are set between adjacent main anchor points in a time-equal division manner, with each sub-point corresponding to a transition stage of the lighting waveform. By comparing the brightness change curve in the light field interferogram, the changing trend of light intensity at each sub-point is determined, and the rising anchor point is set at the point of highest brightness increase, the peak anchor point is set at the inflection point where the brightness change rate changes from positive to negative, and the falling anchor point is set at the point where the brightness decrease rate is most obvious. In this way, a complete lighting change sequence consisting of rising anchor points, peak anchor points, and falling anchor points is formed within the time period of each lighting cycle. Subsequently, the anchor point groups of adjacent cycles are joined end-to-end in chronological order to form a continuous time anchor point sequence from the beginning to the end of illumination. Through this process, the time anchor points not only record the occurrence time of the brightness peak, but also reflect the changing trend of illumination intensity throughout the cycle, providing a precise time reference for the extraction of phase-displaced frames.

[0030] After establishing the time anchor sequence, the corresponding set of image frames is extracted from the actual sampling sequence around each anchor point to identify the phase shift phenomenon between illumination flicker and image sampling rhythm. Each time anchor point represents a key moment in the illumination change process, and when the camera shutter's exposure time window is not perfectly synchronized with these anchor points, brightness shifts will appear in the image. This step uses each time anchor point as the center, expanding forward and backward by a fixed number of image frame intervals to construct a frame sequence window centered on the time anchor point. Within this window, the trend of brightness change over time is observed frame by frame: when the brightness reaches its peak before the time anchor point, it indicates that the shutter exposure is ahead of the illumination peak, resulting in a positive phase shift; when the brightness reaches its peak after the time anchor point, it indicates that the exposure is behind the illumination peak, resulting in a negative phase shift. The shift type, shift direction, and time difference of each frame are recorded using the time anchor point number as an index. For frames that repeatedly exhibit shift phenomena within multiple consecutive cycles, the periodic distribution pattern of shifted frames can be discovered through time comparison. When misalignment occurs repeatedly in multiple adjacent anchor point segments, these frames are grouped into a phase misalignment frame set. Each phase misalignment frame records its anchor point number, phase offset direction, offset time difference, brightness change trend, and corresponding image sampling time, thus forming a detailed record of the misalignment relationship between illumination flicker and exposure rhythm on the time axis.

[0031] After obtaining the set of phase misalignment frames, all frames in the set are organized and archived in chronological order to generate a misalignment frame list describing the temporal distribution characteristics of illumination flicker. Specifically, the process is as follows: First, all phase misalignment frames are arranged from earliest to latest sampling time, establishing a time index. Then, based on the illumination driving cycle, the time axis is divided into multiple equal-length intervals, and phase misalignment frames belonging to the same cycle are grouped together. Within each group, the number, duration, average brightness difference, and order of occurrence of positive and negative phase misalignment frames within that cycle are recorded. These data clearly describe the misalignment rhythm between illumination fluctuations and exposure sampling within each illumination cycle. Subsequently, the misalignment frame groups from each cycle are connected in chronological order to form a complete temporal misalignment chain. In this misalignment chain, the continuous change in the brightness offset direction reflects the phase drift trend of illumination flicker, while the change in time interval reflects the stability of the flicker rhythm. The final misalignment frame list not only includes the specific time position and offset direction of each misalignment frame but also clearly records the periodic pattern of the misalignment distribution throughout the entire sampling cycle. This misaligned frame list uses time as the main axis and brightness changes as the characteristic, transforming the temporal behavior of illumination flicker into a traceable sequence structure, providing precise time basis for the subsequent temporal control of reflection isolation windows and the generation of purified image streams.

[0032] Based on the list of misaligned frames, a reflection isolation window is set up to adjust the image retrieval order during the bullet impact point recognition process and output a cleaned image data stream to actively isolate optical reflection interference in the visual recognition path. To eliminate the optical reflection effects caused by phase interference between illumination flicker and camera sampling, a reflection isolation window with clear temporal and spatial constraints needs to be set based on the generated list of misaligned frames. Under the influence of the reflection isolation window, the retrieval order of image frames during impact point recognition is precisely adjusted, actively avoiding frames affected by illumination interference. After frame data processing, a cleaned image data stream is output, thus achieving active isolation of optical reflection interference in the visual recognition path. The specific steps are as follows: Based on the temporal distribution information of the misaligned frame list, the temporal range of illumination interference is determined, and the temporal structure of the reflection isolation window is established. The misaligned frame list records the temporal position, brightness offset direction, offset magnitude, and corresponding illumination cycle number of each misaligned frame. To ensure that the reflection isolation window can completely cover all frames affected by illumination interference in time, this step first reads the temporal coordinates of all misaligned frames in the list, and groups frames with a time difference less than the shutter exposure cycle of adjacent misaligned frames into the same interference group. For each interference group, the earliest occurrence time is extracted as the front boundary, and the last end time is used as the back boundary. Considering that the brightness change of illumination flicker is not instantaneous, the buffer time needs to be extended based on the front and back boundaries. When the brightness rise rate of the previous frame is higher than the average exposure change rate, the front boundary is extended forward by two sampling intervals; when the brightness decay rate of the next frame is lower than the normal decay rate, the back boundary is extended backward by three sampling intervals. The interval formed after the extension is defined as the temporal range of the reflection isolation window. Each window is marked with a specific start and end time to ensure that the interference area can be clearly identified on the time axis. All windows are arranged in chronological order to form the temporal isolation framework of the entire video sequence.

[0033] After the temporal isolation framework is determined, the reflection isolation window is mapped onto the image frame sequence to obtain the spatial distribution range of the interfering frames. Each frame in the image frame sequence has a precise timestamp and spatial pixel array. The timestamp is used to match the reflection isolation window, and the spatial pixel array is used to determine the brightness distribution area. This step identifies all frames whose times fall within the window range as interfering frames by comparing the timestamps of the image frames with the time intervals of the reflection isolation window. For each interfering frame, the spatial coordinate data of the corresponding time period in the transient reflection seed set is read to determine the distribution location of light reflection within that frame. For each reflection seed, its spatial coordinate center, boundary range, and brightness peak are extracted, and these data are projected onto the frame image to mark the specific area of ​​the interfering light spot. Subsequently, all light spot areas within the same time window are merged to form a spatial interference area, and its range is represented by rectangular coordinates on the image. The pixel positions of its upper left and lower right corners, the brightness distribution gradient, and the number of consecutive frames are recorded. After this processing, each time window corresponds to a specific spatial interference area, constituting the spatial constraint structure of the reflection isolation window. In this way, the reflection isolation window has both a start and end boundary in the time dimension and a coverage area in the spatial dimension, realizing the definition of interference in both the time and space domains.

[0034] After the spatiotemporal structure of the reflection isolation window is established, the image frame retrieval order in the impact point recognition process is adjusted to avoid frames affected by illumination interference. This step first reads the frame sequence to be processed in the recognition process and checks whether the timestamp of each frame falls within the time range of any reflection isolation window. If the timestamp of a frame is within the window's time interval, it is determined whether its pixel coordinates overlap with the corresponding spatial interference area. If both temporal and spatial conditions are met, the frame is identified as an interference frame and removed from the recognition sequence. For removed frames, their temporal position and the number of adjacent normal frames are recorded for time compensation during subsequent data reassembly. To maintain the continuity of the recognition rhythm, transition frame information is inserted temporally, based on two adjacent normal frames. The brightness value of the transition frame is determined by the average brightness of the preceding and following frames, and its temporal position is located at the midpoint between the two frames, used to create a smooth brightness transition in the image stream. Subsequently, the remaining normal frames are rearranged in chronological order to form an updated recognition frame sequence. By using this sequential adjustment and transitional frame interpolation method, the recognition process avoids all interfering time periods and spatial areas, thereby ensuring that the image data used in the bullet impact point recognition process all come from valid images under real exposure conditions.

[0035] After adjusting the image frame order, the updated recognition frame sequence is reassembled and output to form a purified image data stream. The frame reassembly stage first checks the inter-frame intervals frame by frame in chronological order to ensure that the time difference between adjacent frames remains within a preset sampling period. If the time interval between adjacent frames is too large, a compensation frame is inserted with the end time of the previous frame and the start time of the next frame as the boundary to maintain a uniform and continuous timeline. Subsequently, the brightness distribution of all frames is smoothed. In the pixel array of each frame, the brightness difference at the same pixel position in adjacent frames is used as a smoothing parameter to adjust the brightness of pixels with abrupt changes, thereby avoiding image jumps caused by brightness differences between frames. The reassembled and smoothed frame sequence is continuously output to form a purified image data stream. In this data stream, all frames within the time and spatial range of the reflection isolation window have been completely removed or replaced, and the brightness changes between the remaining frames maintain a natural transition, resulting in a visually stable and coherent temporal rhythm for the entire image. The purified image data stream is input into the subsequent visual recognition process. In this data stream, the optical reflection signals caused by lighting flicker interference have been effectively isolated, and the changes in image brightness completely correspond to the actual impact point event, ensuring the accuracy of recognition and the consistency of judgment time.

[0036] Dual-end dynamic control is performed in the purified image data stream. By synchronously controlling the phase breathing slide of the lamp driver end and the micro-step movement of the camera shutter end, a flash avoidance corridor is formed, so that false impact points are actively extinguished and the hit recognition rhythm is kept stable. To completely eliminate residual light flicker interference in the purified image data stream and maintain a stable temporal rhythm in the hit recognition process, a dynamic collaborative control mechanism needs to be established between the lighting driver and the image acquisition end. By implementing phase breathing shift at the lighting driver end and micro-stepping at the camera shutter end, the working rhythms of both ends are controllably misaligned on the time axis, thus constructing a flicker avoidance corridor. This ensures that the light peak and the exposure window remain non-overlapping, thereby actively extinguishing false impact points and maintaining a stable recognition rhythm. The specific steps are as follows: After the purified image data stream output stabilizes, the light emission control signal from the lamp driver and the exposure trigger signal from the camera shutter are synchronously acquired to establish a time reference for dual-end dynamic control. The light emission control signal from the lamp driver is a periodic current waveform. The rising edge of this waveform represents the rapid increase phase of light intensity, the peak segment represents the stable illumination phase, and the falling edge represents the brightness decay phase. The exposure trigger signal from the camera shutter is a periodic square wave, and the shutter opening time represents the exposure duration. To establish the relative time relationship between the two, this step synchronously samples the light emission signal and the exposure signal under the same clock source and records the time deviation between each pair of signals. Specifically, the start time, peak time, and end time of the lamp's light emission cycle, as well as the specific time points when the camera shutter opens and closes, are recorded. By continuously acquiring data from multiple cycles, a correspondence between the illumination light emission cycle and the shutter exposure cycle is obtained. For each cycle, the time difference between the time point when the light peak occurs and the center time of the exposure window is recorded; this time difference is the phase offset. The distribution of phase offset can intuitively reflect the overlap trend between the illumination peak and the exposure window in time, providing basic data for subsequent dynamic control.

[0037] After obtaining the dual-end phase offset data, phase breathing slip is executed based on the waveform control signal at the lamp driver end. This causes a periodic, controllable shift in the illumination peak time on the time axis, achieving active avoidance of the illumination rhythm. The phase breathing slip process refers to making a small time adjustment to the start time of each illumination cycle, causing the illumination peak position to move slowly in continuous cycles. Based on the phase offset direction and amplitude recorded in the previous step, when the illumination peak appears within the shutter exposure window, the start time of the illumination cycle is delayed by a fixed time interval, causing the illumination peak to slide backward on the time axis; when the illumination peak lags behind the exposure window, the start time of the illumination cycle is advanced by the same time interval, causing the peak to slide forward on the time axis. To maintain the stability of the illumination rhythm, the time amplitude of each slip is controlled within one-hundredth of the illumination cycle, and the slip direction remains consistent for ten consecutive cycles, causing the illumination peak to form a regular breathing-like movement. After multiple cycles of slipping, the peak of the illumination waveform no longer overlaps with the exposure window, but maintains a safe interval on the time axis. This breathing slip creates an adaptive adjustment process, enabling the lighting rhythm to dynamically match the camera sampling rhythm, providing a time drift basis for the establishment of the flash avoidance corridor on the light source side.

[0038] After the lighting fixtures establish a breathing-like sliding rhythm, to maintain synchronized flash avoidance at both ends, micro-stepping control is applied to the exposure rhythm at the camera shutter end. This causes a slight displacement of the sampling window on the time axis, offsetting it from the light peak and maintaining dynamic matching. The shutter micro-stepping process involves adjusting the shutter opening time or scaling the exposure duration to fine-tune the camera's exposure range as the light peak moves. This step uses the real-time phase information of the lighting fixture's emission signal as a reference, calculating the time difference between the center moment of the exposure window and the light peak moment within each sampling cycle. When the light peak enters the exposure window earlier, the shutter opening time is delayed by a fixed time interval, causing the sampling window to slide backward; when the light peak lags behind the exposure window, the shutter opening time is advanced by the same time interval, causing the sampling window to slide forward. To prevent brightness fluctuations caused by unstable exposure time, the micro-stepping amplitude is controlled within one-tenth of the exposure time length, and a periodic symmetrical adjustment method is used, alternating the shifting direction in multiple consecutive sampling cycles. Through this micro-stepping motion, the camera's sampling window is continuously fine-tuned along the timeline, ensuring that it completes exposure within the low-brightness range of the illumination cycle, thereby preventing high-brightness peaks from entering the sampling window range. This process achieves dynamic tracking of the sampling end's movement to the light source end, creating a controllable avoidance relationship between the two in time.

[0039] Under the combined effect of phase breathing slip at the lamp driver end and micro-stepping at the camera shutter end, a dynamic flicker avoidance corridor is formed on the time axis, and false impact points are actively extinguished by purifying the image data stream. The flicker avoidance corridor refers to a continuous, non-overlapping interval formed between the illumination period and the sampling period, which moves synchronously with changes in illumination and sampling rhythms. This step aligns the phase information at both ends in time, establishing a stable gap between the falling segment of the illumination waveform and the rising segment of the sampling window; this gap is the time range of the flicker avoidance corridor. Once the flicker avoidance corridor is formed, the distance between the illumination peak and the exposure period remains constant, and illumination interference no longer enters the exposure channel during imaging. Because illumination interference is actively avoided, the brightness distribution in the purified image data stream returns to the real scene illumination state, and all bright spots caused by flicker interference no longer appear in the image. As the illumination peak and exposure window continue to be misaligned, false impact points are gradually extinguished, and the recognition system only responds to true hit events. Through this process, the entire imaging channel maintains a stable rhythm in time and a balanced brightness in space, thus ensuring that the timing rhythm of target recognition is consistent with the triggering cycle of the feedback signal, and ensuring that the judgment results of the interactive target shooting game are accurate and reliable under dynamic lighting conditions.

[0040] This invention establishes a dual-end dynamic control mechanism between the illumination driving end and the image acquisition end, creating a continuously staggered anti-flash corridor between the illumination emission cycle and the camera sampling cycle on the time axis, fundamentally eliminating transient reflection interference under high-frequency flash illumination conditions. Dynamic separation of the illumination peak and exposure window is achieved through phase breathing slip at the lamp driving end and micro-stepping at the camera shutter end. False impact points are actively extinguished during imaging, restoring stable image brightness distribution and ensuring the purity and continuity of the target shooting image data stream during the sampling phase, providing a reliable visual input basis for the judgment process.

[0041] This invention achieves stable and consistent target recognition rhythm by continuously implementing dual-end dynamic control within the purified image data stream, automatically maintaining phase balance between the illumination rhythm and the sampling rhythm during operation. As the lightning avoidance corridor forms and persists, image brightness fluctuations are suppressed, the triggering sequence of the recognition signal remains constant, and the rhythmic stability and response accuracy of the judgment feedback process are simultaneously improved. Through this control method, interactive target shooting games can still achieve a smooth recognition experience and consistent interactive feedback even in complex lighting environments.

[0042] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multimodal perception-based interactive shooting game judgment and feedback method, characterized in that, Includes the following steps: The driving frequency of the lighting fixtures and the shutter frequency of the camera are collected, the frequency intersection area of ​​the two is determined, and a light field interferogram is generated to visualize the interference relationship between the illumination rhythm and the image sampling rhythm. High-brightness burst trajectories are extracted based on light field interferograms, and a set of transient reflection seeds is constructed to define the spatiotemporal distribution range of abnormal light spots caused by illumination flicker. A time anchor sequence is established around the transient reflection seed set, phase misalignment frames are extracted, and a list of misalignment frames is generated; Based on the list of misaligned frames, a reflection isolation window is set up to adjust the image retrieval order during the bullet impact point recognition process, output a clean image data stream, and actively isolate optical reflection interference in the visual recognition path; Dual-end dynamic control is performed in the purified image data stream. By synchronously controlling the phase breathing slide of the lamp driver end and the micro-step movement of the camera shutter end, a flash avoidance corridor is formed, so that false impact points are actively extinguished and the hit recognition rhythm is kept stable.

2. The multimodal perception-based interactive shooting game judgment and feedback method according to claim 1, characterized in that, The process of generating the light field interferogram is as follows: The light emission control signal from the driver end of the lighting device and the exposure trigger signal from the shutter end of the image acquisition device are collected, and two sets of time series data are recorded under a unified time reference, so that the waveform of the change in lighting brightness and the exposure opening curve are kept synchronized on the time axis. Based on the synchronized time series, determine the intersection zone of the illumination brightness change cycle and the exposure time window on the time axis, record the start time, end time and overlap duration of each intersection zone, and form the coupling interval between the illumination rhythm and the sampling rhythm. The illumination intensity time curve and the exposure curve are mapped onto the spatial plane according to the time correspondence to generate the light field interference distribution structure; The brightness of the light field interference distribution structure is expanded in time and extended in space to form a three-dimensional light field interference spectrum, so that the interference relationship between the illumination rhythm and the sampling rhythm can be presented in a visual way.

3. The multimodal perception-based interactive shooting game judgment and feedback method according to claim 2, characterized in that, The steps to construct a transient reflection seed set are as follows: After the light field interferogram is generated, the brightness change value is read line by line along the time axis, the time periods of brightness increase and decrease are recorded, the high brightness burst segments are identified and their spatial coverage is marked. Extend and connect consecutive or overlapping bright burst segments in spatial coordinates within adjacent time periods to form a set of bright burst trajectories spanning multiple sampling periods; The instant with the fastest increase in brightness and the highest peak value is extracted along the direction of the high brightness burst trajectory as the transient reflection candidate point, and the time position, spatial position, brightness peak value and duration are recorded to form a transient reflection seed set; Transient reflection seeds are merged sequentially in the time dimension and aggregated according to proximity in the spatial dimension to generate an abnormal light spot distribution map.

4. The multimodal perception-based interactive shooting game judgment and feedback method according to claim 3, characterized in that, When generating the abnormal light spot distribution map, the center position and boundary contour of the reflection area unit are determined by comprehensively weighting the brightness peak, duration and spatial diffusion range of the transient reflection seed. This ensures that the time span and spatial concentration of each abnormal light spot in the reflection distribution map are consistent, thereby improving the accuracy and stability of the spatiotemporal distribution definition of abnormal light spots.

5. The multimodal perception-based interactive shooting game judgment feedback method according to claim 3, characterized in that, The steps to generate a list of misaligned frames are as follows: After the transient reflection seed set is established, the time coordinates of all seeds are sorted and a continuous time node chain is formed. Based on the driving cycle of the lighting device, the node with the largest brightness peak is extracted to establish a time anchor sequence. Rising anchors, peak anchors, and falling anchors are set between adjacent time anchors to fully record the changing trend of light intensity within one cycle, forming a continuous time anchor sequence; Extract the corresponding image frame interval with the time anchor point as the center, analyze the time position of the brightness peak, identify positive phase misalignment and negative phase misalignment, and form a set of phase misalignment frames; The phase misalignment frames are archived in the order of sampling time, and the misalignment direction, duration and brightness difference are recorded to generate a misalignment frame list.

6. The multimodal perception-based interactive shooting game judgment and feedback method according to claim 5, characterized in that, The distribution of each time anchor point in the time anchor point sequence is determined according to the rate of change of illumination brightness. The rising anchor point corresponds to the point where the brightness growth rate is the highest, the peak anchor point corresponds to the turning point where the brightness change changes from increasing to decreasing, and the falling anchor point corresponds to the point where the brightness decay rate is the largest, so that the intensity change process within the illumination cycle can be completely marked on the time axis.

7. The multimodal perception-based interactive shooting game judgment and feedback method according to claim 6, characterized in that, The steps for outputting the purified image data stream are as follows: Based on the temporal distribution information of the misaligned frame list, the temporal range of the lighting interference is determined, a reflection isolation window with front and back boundaries is established, and a buffer time is set on the boundary to form a temporal isolation framework. The reflection isolation window is mapped onto the image frame sequence. By comparing the timestamp with the spatial coordinate data in the transient reflection seed set, the spatial coverage of the illumination reflection is determined, forming a reflection isolation window structure with spatiotemporal constraints. During the bullet impact point recognition process, the image frame retrieval order is adjusted according to the reflection isolation window. Interference frames that fall into the window in both time and space are removed and transition frame information is inserted to maintain the continuous recognition rhythm. The updated recognition frame sequence is time-smoothed and brightness-compensated for recombination, and the cleaned image data stream is output.

8. The multimodal perception-based interactive shooting game judgment feedback method according to claim 7, characterized in that, The temporal boundary of the reflection isolation window is adaptively adjusted according to the rate of brightness change. When the rate of brightness increase is higher than the average rate of exposure change, the front boundary extends forward, and when the rate of brightness decay is lower than the normal rate of decrease, the rear boundary extends backward.

9. The multimodal perception-based interactive shooting game judgment feedback method according to claim 7, characterized in that, The following steps are taken to create a flash avoidance corridor on the timeline by synchronously controlling the phase breathing slide of the lamp driver end and the micro-step movement of the camera shutter end in the purified image data stream: After the purified image data stream output is stabilized, the light emission control signal of the lamp driver end and the exposure trigger signal of the camera shutter end are synchronously acquired to record the phase offset between the peak time of illumination and the center time of the exposure window, and to establish a time reference for dual-end dynamic control. The timing of the start of the light emission cycle at the lamp driver end is controlled by the phase shift direction and amplitude, so that the light peak moves periodically on the time axis, forming a phase breathing shift. Using the real-time phase information of the lamp's light emission signal as a reference, the sampling window is exposed within the low-brightness range of the illumination cycle by adjusting the shutter opening time or exposure duration. Synchronize and align the timing information of the lamp phase breathing slide with the shutter microstep movement to form a flash avoidance corridor between the falling segment of the illumination waveform and the rising segment of the sampling window.

Images