Visual interaction method of splicing processor

Abstract

The invention discloses a visual interaction method for splicing processors, and relates to the technical field of man-machine interaction, and the method comprises the steps: tracking a user iris motion track in real time, determining a processor in a current view cone focus region, and marking a processor in a non-focus region outside the view cone focus region; when the node position similarity exceeds a preset verification threshold value, triggering a configuration change command corresponding to execution of the prediction instruction to a marked non-focus area processor; new load data generated by the configuration change command after execution are collected in real time, whether the new load data exceed the preset load threshold value or not is judged again, and a dynamic monitoring and interactive closed loop is formed. According to the method, the pulse spectrum is projected to the peripheral visual field of the retina of a user, a non-intrusive optical signal is formed in the peripheral visual field of the retina by using the infrared spectrum, and the load alarm is transmitted on the premise of not interfering central focus processing of cone cells by using the high sensitivity of the rod cells to a dynamic light source.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a visual interaction method for a splicing processor. Background Technology

[0002] Eye-tracking technology has been commercially applied in industrial control. It primarily uses infrared light to reflect iris feature points, combined with IMU attitude compensation, to calculate gaze direction and improve positioning accuracy. Simultaneously, distributed processor load monitoring can collect CPU, memory, and network metrics in real time via the SNMP protocol. Combined with augmented reality technology, visual status indicators can be overlaid on the surface of physical devices. Existing solutions can already achieve basic eye-tracking interaction and load alarm functions, providing a technological foundation for complex control environments.

[0003] Existing technologies have shortcomings in handling anomalies in non-focus areas. Traditional screen pop-up warnings forcibly shift the user's focus, causing interruptions to the main task; while static physical indicator lights cannot accurately convey load levels and operational instructions, requiring users to switch their attention to interpret alarm information. This intermittent interaction will cause response delays in scenarios with continuous focus tasks, such as air traffic control and industrial production lines, becoming a key bottleneck for high-load human-machine collaboration. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a visual interaction method for splicing processors to solve the problems of traditional warning methods interrupting the user's focus on the task and the lack of accurate operation guidance.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a visual interactive method for stitching processors, which includes tracking the user's iris movement trajectory in real time, determining the processors within the current focal region of the visual cone, and marking the processors in non-focal regions outside the focal region of the visual cone. When the load data of the processor in the marked non-focal area exceeds a preset load threshold, a pulse spectrum is projected onto the user's peripheral visual field. Based on the pulse spectrum, a behavior prediction model is constructed by calling the user's historical operation sequence, and a prediction instruction is generated. A semi-transparent operation guidance ghost image is rendered on the physical device surface of the processor in the non-focal area. The system continuously captures the actual skeletal nodes of the user's operation using computer vision, and calculates the similarity between the node positions of the actual skeletal nodes and the semi-transparent operation guide phantom. When the node location similarity exceeds the preset verification threshold, the configuration change command corresponding to the prediction instruction is triggered and sent to the marked non-focus area processor. Real-time collection of new load data generated after the execution of configuration change commands, re-determination of whether the preset load threshold is exceeded, forming a dynamic monitoring and interactive closed loop.

[0007] As a preferred embodiment of the visualization interaction method for the stitching processor described in this invention, the specific steps of the processor that simultaneously marks the non-focal area outside the focal region of the view frustum are as follows: The three-dimensional coordinate set of the iris reflection spot at the current moment is obtained by using a dual-band infrared ring light source, and the Euclidean displacement vector of the three-dimensional coordinate set of the iris reflection spot at the previous moment is calculated. Based on the predefined displacement-angle response mapping table, the Euclidean displacement vector is converted into the line-of-sight correction angle, and the original line-of-sight direction angle is calculated by combining the fixed light source-camera position vector of the device. The original line-of-sight angle is input into the head-mounted device's inertial measurement, and then transformed by the attitude rotation matrix to generate a three-dimensional line-of-sight vector in the world coordinate system; When the angle between the 3D view vector and the processor's spatial position vector is less than the preset focus threshold, the processor is determined to be within the current view frustum focus area; otherwise, it is marked as a non-focus processor.

[0008] As a preferred embodiment of the visualization interaction method of the stitching processor described in this invention, the specific steps of projecting pulse spectrum into the peripheral visual field of the user's retina are as follows: Collect load data of the processors in the marked non-focus areas and calculate the comprehensive load index of the processors in the non-focus areas; When the overall load index of the non-focus area processor exceeds the preset load threshold, the load feature vector is extracted from the load data, and the spatial coordinate position of the non-focus area processor and the user's eye coordinate position are obtained at the same time. Based on the three-dimensional gaze vector and the user's eye coordinate position, the mapping coordinates of the spatial coordinate position of the non-focal area processor in the peripheral visual field of the retina are obtained; Based on the load feature vector, pulse spectral parameters are generated, and the pulse spectrum corresponding to the pulse spectral parameters is projected at the mapping coordinates of the peripheral visual field of the retina.

[0009] As a preferred embodiment of the visualization interaction method for the splicing processor described in this invention, the specific steps for generating prediction instructions are as follows: Analyze the pulse spectrum to extract wavelength, frequency and light intensity data. Determine the processor load type in the non-focal area based on the wavelength data, determine the load urgency level based on the frequency data, and generate load characteristic identifiers. Retrieve the user's historical operation sequence, perform spatiotemporal weighted filtering of historical operation actions based on load characteristic identifiers, and output the weighted historical operation sequence; A behavior prediction model is constructed based on a weighted historical operation sequence. The hierarchical attenuation factor of the behavior prediction model is dynamically adjusted according to the light intensity data to generate a modulated behavior prediction model. The weighted historical operation sequence is input into the modulated behavior prediction model to generate a prediction instruction set. The confidence level of the prediction instruction set is calculated. When the confidence level exceeds the preset confidence threshold, the prediction instruction is output.

[0010] As a preferred embodiment of the visualization interaction method for the splicing processor described in this invention, the specific steps of rendering the semi-transparent guided shadow are as follows: Parse the semantic structure of the predicted instruction and generate an action descriptor containing the target position and action parameters; Based on the target location, construct a three-dimensional spatial model of the surface of the corresponding non-focal region processor; Based on the motion parameters, the ghost intensity field is calculated, and the ghost intensity distribution field is generated on the surface three-dimensional space model; The system acquires real-time user pupil diameter data and modulates the transparency parameters of the ghost image intensity distribution field based on the user pupil diameter data to generate a transparency-modulated ghost image field. The virtual image field after transparency modulation is converted into a holographic phase map, and a semi-transparent operation guide virtual image is projected onto the surface of the physical device through a spatial light modulator.

[0011] As a preferred embodiment of the visualization interaction method for the splicing processor described in this invention, the specific steps for calculating the similarity between the node positions of the semi-transparent operation-guided phantom are as follows: By fusion of multiple sensors to capture the user's actual operation skeletal node data, a spatiotemporal sequence of skeletal motion is generated. Optical reconstruction is then performed on the semi-transparent operation-guided ghost image to generate a spatiotemporal sequence of ghost image skeletal motion. Based on the gradient change characteristics of the spatiotemporal sequence of skeletal motion, user motion intent feature data is generated; based on the gradient change characteristics of the spatiotemporal sequence of ghost skeleton motion, ghost motion intent feature data is generated. Calculate the node position similarity between the user's motion intention feature data and the ghost image's motion intention feature data.

[0012] As a preferred embodiment of the visual interaction method for the splicing processor described in this invention, the specific steps for sending the configuration change command to the marked non-focus area processor are as follows: Based on node position similarity, spatiotemporal sequence of skeleton motion, and spatiotemporal sequence of phantom skeleton motion, generate operation intent. Figure 1 Coherence vector; Operational intention Figure 1 The consistency vector is used to perform spatiotemporal integration calculation and output dynamic confidence score. When the dynamic confidence score exceeds the preset verification threshold, an encrypted execution protocol packet is generated to encapsulate the configuration change command. Send encrypted execution protocol packets to the marked non-focus area processor.

[0013] As a preferred embodiment of the visualization and interaction method for the splicing processor described in this invention, the specific steps for forming a dynamic monitoring and interaction closed loop are as follows: Collect new load data after the processor in the marked non-focus area executes the configuration change command, and recalculate the new comprehensive load index; When the new comprehensive load index exceeds the preset load threshold, a pulsed red spectrum is projected onto the user's retina; when the new comprehensive load index does not exceed the preset load threshold, a steady-state green spectrum is projected. Track the user's iris movement trajectory. When the iris moves into the pulsed red spectrum region, generate a load migration command. When the iris gazes into the steady green spectrum region for more than a preset time, generate a configuration maintenance command. After executing the load migration command or configuration maintenance instruction, new load data is collected again for judgment until the comprehensive load index does not exceed the preset load threshold.

[0014] The beneficial effects of this invention are as follows: By projecting pulsed spectra into the peripheral visual field of the user's retina, a non-invasive light signal is formed in the peripheral visual field using infrared spectroscopy. Utilizing the high sensitivity of rod cells to dynamic light sources, load alarms are transmitted without interfering with the central focal processing of cone cells. The pulse frequency is linearly adjusted according to the load exceeding the threshold amplitude, allowing the cerebral cortex to determine the abnormality level through frequency analysis. Simultaneously, the spectral projection position is bound to the spatial coordinates of the target processor, providing a spatial reference for operation guidance. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart for the visual interaction method of splicing processors.

[0017] Figure 2 A flowchart for defining the focus area and marking the non-focus areas.

[0018] Figure 3 This is a flowchart of pulsed spectral projection.

[0019] Figure 4 This is a flowchart for guiding the rendering of phantom images and calculating similarity. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a visual interactive method for a splicing processor, comprising the following steps: S1. Track the user's iris movement trajectory in real time, determine the processors within the current focal area of ​​the visual cone, and mark the processors in the non-focal areas outside the focal area of ​​the visual cone.

[0024] S1.1: Furthermore, the three-dimensional coordinate set of the iris reflection spot at the current moment is obtained by using a dual-band infrared ring light source, and the Euclidean displacement vector of the three-dimensional coordinate set of the iris reflection spot at the previous moment is calculated. Specifically, a dual-band infrared ring light source alternately projects infrared beams onto the iris surface, and a light field camera simultaneously captures a sequence of iris reflection spot images. Differential noise reduction is performed on the images of the two different wavelengths to extract the pixel coordinates of the effective spot center point. Combined with the depth mapping matrix pre-calibrated by the light field camera and the geometric parameters of the light source, a three-dimensional spatial coordinate set of the spot is obtained. The cached three-dimensional coordinate set of the iris reflection spot from the previous moment is called, and the Euclidean distance of the corresponding spot is obtained point by point.

[0025] It should be noted that the effective spot center pixel coordinates refer to the geometric center position coordinates of the iris reflection spot on the image sensor after dual-band differential noise reduction processing; The pre-calibrated depth mapping matrix refers to the transformation relationship matrix from pixel coordinates to three-dimensional spatial depth established during the factory calibration stage of the light field camera; The pre-calibrated light source geometry parameters refer to the fixed spatial position and angle parameters between the infrared ring light source array and the light field camera lens.

[0026] S1.2: Based on the predefined displacement-angle response mapping table, the Euclidean displacement vector is converted into the line-of-sight correction angle, and the original line-of-sight direction angle is calculated by combining the fixed light source-camera position vector of the device. It should be noted that the predefined displacement-angle response mapping table refers to the lookup table of the correspondence between the magnitude and direction of the Euclidean displacement vector and the correction angle of the line of sight direction, which was established through eye movement calibration experiments.

[0027] Specifically, the predefined displacement-angle response mapping table is queried, the magnitude and direction index value of the Euclidean displacement vector are matched, the viewing direction correction angle is output, the fixed light source-camera position vector of the device is read, the fixed light source-camera position vector of the device describes the fixed spatial vector from the center point of the light source array to the optical center of the camera, the viewing direction correction angle is used as an offset, and superimposed on the reference angle of the line connecting the light source-camera position vector and the center point of the eyeball, and the original viewing direction angle of the main visual axis of the eyeball relative to the device coordinate system is obtained through the angle in the eyeball coordinate system.

[0028] S1.3: Input the original line of sight direction angle into the head-mounted device's inertial measurement, and generate a three-dimensional line of sight vector in the world coordinate system through attitude rotation matrix transformation; Specifically, the original line-of-sight angle is input into the head-mounted device's inertial measurement unit. The head-mounted device's inertial measurement unit outputs the device's attitude quaternion in real time. The device's attitude quaternion is converted into an attitude rotation matrix. The horizontal azimuth and vertical pitch components of the original line-of-sight angle are read. An initial line-of-sight vector in spherical coordinates is constructed in the device's local coordinate system. The attitude rotation matrix is ​​applied to perform a rotation transformation on the initial line-of-sight vector to obtain a three-dimensional line-of-sight vector in the world coordinate system.

[0029] S1.4: When the angle between the three-dimensional view vector and the processor's spatial position vector is less than the preset focus threshold, the processor is determined to be within the current view frustum focus area; otherwise, it is marked as a non-focus processor.

[0030] It should be noted that the preset focus threshold is set based on the physiological limit of the human eye's focusing cone angle, and the example value is 5.

[0031] Specifically, the cosine of the angle between the 3D view vector and the processor's spatial position vector is obtained. The cosine of the angle is compared with the cosine of a preset focus threshold. When the cosine of the angle is less than the cosine of the preset focus threshold, the processor is determined to be located within the current view frustum focus area. When the cosine of the angle is greater than or equal to the cosine of the preset focus threshold, the processor is marked as a non-focus processor.

[0032] S2. When the load data of the processor in the marked non-focal area exceeds the preset load threshold, a pulse spectrum is projected onto the peripheral visual field of the user's retina.

[0033] S2.1: Further, collect the load data of the marked non-focus area processors and calculate the comprehensive load index of the non-focus area processors; Specifically, the CPU utilization percentage data of non-focus area processors is collected through the processor's built-in performance counter, the memory utilization percentage data is collected through the processor memory management, the network bandwidth utilization percentage data is collected through the processor network interface controller, and the core temperature data in degrees Celsius is collected through the processor temperature sensor. The CPU utilization percentage data, memory utilization percentage data, network bandwidth utilization percentage data, and core temperature data in degrees Celsius are then combined to generate load data.

[0034] The overall load index of the non-focus region processor is calculated using the following expression: ; In the formula, This represents the overall load index of the non-focus area processor. Indicates the integration time window. This represents the reciprocal of the integration time window. Indicates from time arrive definite integral, Indicates the current moment. Indicates the starting time of integration. Represented as Non-focus area processor utilization at any given moment This indicates the processor baseline utilization in the non-focus area. Indicates the memory sensitivity coefficient. Indicates memory usage. This indicates the rate of change in memory usage. This represents the exponentially decaying weight term. Indicates the time decay factor. Represents the infinitesimal element at a given moment.

[0035] It should be noted that the memory sensitivity coefficient is determined through regression analysis of historical load data and is used to quantify the contribution weight of the memory change rate to the overall load. The example value is 0.8. The time decay factor is calculated based on the human-computer interaction response delay model and controls the decay rate of historical load data. The example value is 0.1.

[0036] S2.2: When the overall load index of the non-focus area processor exceeds the preset load threshold, extract the load feature vector from the load data, and at the same time obtain the spatial coordinate position of the non-focus area processor and the user's eye coordinate position. It should be noted that the preset load threshold is set based on the processor's safe operating temperature and performance saturation threshold; the example value is 0.85.

[0037] Specifically, when the overall load index of the non-focus area processor does not exceed the preset load threshold, the current configuration is maintained and load data is collected periodically. When the overall load index of the non-focus area processor exceeds the preset load threshold, a load feature vector extraction operation is performed. The CPU utilization percentage data, memory utilization percentage data, network bandwidth utilization percentage data, and core temperature data are separated from the load data and combined into a load feature vector. The spatial coordinate position storage register of the non-focus area processor is accessed, the spatial coordinate position is read, and the user's eye coordinate position output by the eye tracking of the head-mounted device is called.

[0038] S2.3: Based on the three-dimensional gaze vector and the user's eye coordinate position, obtain the mapping coordinates of the spatial coordinate position of the non-focal area processor in the peripheral visual field of the retina; Specifically, the spatial direction vector pointing from the user's eye coordinate position to the non-focal area processor spatial coordinate position is obtained. The angle between the spatial direction vector and the three-dimensional line-of-sight vector is measured in radians. Based on the physiological structural characteristics of the eye, the angle in radians is converted into retinal spherical coordinate parameters. Referring to the limitations of the projection range of the peripheral visual field of the retina, a radial distance scaling operation is performed on the spherical coordinate parameters. The scaled horizontal azimuth angle parameters and vertical pitch angle parameters are output as the mapping coordinates of the peripheral visual field of the retina.

[0039] It should be noted that the reference to the projection range limitation of the peripheral visual field of the retina refers to the boundary constraints of the maximum projection angle in the horizontal direction and the maximum projection angle in the vertical direction, which are determined based on the distribution density of rod cells and the location of the visual blind spot.

[0040] S2.4: Generate pulse spectral parameters based on the load feature vector, and project the pulse spectrum corresponding to the pulse spectral parameters at the mapped coordinates of the peripheral visual field of the retina.

[0041] Specifically, the CPU utilization percentage component, memory utilization percentage component, network bandwidth utilization percentage component, and core temperature in Celsius component are separated from the load feature vector. The CPU utilization percentage component is mapped to a pulse spectrum wavelength parameter, the memory utilization percentage component is mapped to a pulse spectrum intensity parameter, the network bandwidth utilization percentage component is mapped to a pulse spectrum flicker frequency parameter, and the core temperature in Celsius component is mapped to a pulse spectrum color temperature shift parameter. These components are combined to generate a pulse spectrum parameter set. The horizontal azimuth and vertical pitch components of the retinal peripheral field of view mapping coordinates are retrieved, and the pulse spectrum corresponding to the pulse spectrum parameter set is projected at the specified coordinate orientation in the retinal peripheral field of view through a micro-projection optical engine.

[0042] S3. Based on the pulse spectrum, call the user's historical operation sequence to build a behavior prediction model, generate prediction instructions, and render a semi-transparent operation guidance phantom on the physical device surface of the processor in the non-focus area.

[0043] S3.1: Going further, analyze the pulse spectrum, extract wavelength, frequency and light intensity data, determine the processor load type in the non-focal area based on the wavelength data, determine the load urgency level based on the frequency data, and generate load characteristic identifiers; Specifically, the pulse spectrum projected from the peripheral visual field of the retina is received, and a spectral analyzer is used to decompose the pulse spectrum, extract the center wavelength value, pulse repetition frequency value, and peak light intensity value. A preset wavelength-load type mapping table is consulted to match the non-focal area processor load type classification code corresponding to the center wavelength value. A preset frequency-emergency level mapping table is consulted to match the load emergency level classification code corresponding to the pulse repetition frequency value. The load type classification code and the load emergency level classification code are combined to generate a load feature identifier.

[0044] It should be noted that the preset wavelength-load type mapping table refers to a lookup table that predefined the correspondence between pulse center wavelength values ​​and processor load type classification codes; The preset frequency-emergency level mapping table refers to a lookup table that maps predefined pulse repetition frequency values ​​to load processing emergency level classification codes.

[0045] S3.2: Retrieve the user's historical operation sequence, perform spatiotemporal weighted filtering of historical operation actions based on load characteristic identifiers, and output the weighted historical operation sequence; Specifically, the system accesses the user's historical operation sequence database, retrieves a historical record set containing operation action type, timestamp, and load scenario identifier, parses the load feature identifier, extracts the load type classification code and load urgency level classification code, obtains the absolute value of the time difference between the historical record timestamp and the current time, generates a time weight coefficient, matches the load scenario identifier in the historical record set with the current load type classification code, generates a load matching degree weight coefficient, combines the time weight coefficient and the load matching degree weight coefficient to obtain a comprehensive weight value, and weights the comprehensive weight value to generate a weighted historical operation sequence.

[0046] S3.3: Construct a behavior prediction model based on a weighted historical operation sequence, dynamically adjust the hierarchical attenuation factor of the behavior prediction model according to the light intensity data, and generate a modulated behavior prediction model; It should be noted that the specific process of constructing the behavior prediction model is as follows: the long short-term memory network architecture is initialized with a weighted historical operation sequence. The weighted historical operation sequence contains three-dimensional data of operation action type, timestamp interval and load numerical features. The three-dimensional data is encoded into a fixed-dimensional feature vector sequence. The input feature vector sequence is trained through a sequence-to-sequence training architecture. The network weight parameters are optimized using the time backpropagation algorithm. The output layer uses Softmax activation to generate the probability distribution of predicted operation actions. The output of the trained long short-term memory network architecture is the behavior prediction model.

[0047] Specifically, a weighted historical operation sequence is used to initialize a long short-term memory network architecture as the basic framework of the behavior prediction model. The weighted historical operation sequence contains three-dimensional data of operation action type, timestamp interval and load numerical features. The light intensity data of the pulse spectrum is read, and the hierarchical attenuation factor adjustment coefficient is output through a preset light intensity-attenuation mapping. The hierarchical attenuation factor adjustment coefficient is written into the hidden state transit path parameters of the basic framework of the behavior prediction model. The updated hidden state transit path parameters generate the modulated behavior prediction model.

[0048] It should be noted that the preset light intensity-attenuation mapping refers to the correspondence table between the pre-calibrated peak light intensity value of the pulse spectrum and the adjustment coefficient of the attenuation factor of the long short-term memory network hierarchy.

[0049] S3.4: Input the weighted historical operation sequence into the modulated behavior prediction model to generate a prediction instruction set, calculate the confidence level of the prediction instruction set, and output the prediction instruction when the confidence level exceeds the preset confidence threshold.

[0050] Specifically, the weighted historical operation sequence is input into the modulated behavior prediction model, forward propagation is performed, and a prediction instruction set is generated. The prediction instruction set contains the probability value of each operation instruction. The confidence level of the prediction instruction set is calculated, and the relationship between the confidence level and the preset confidence threshold is compared. When the confidence level is greater than the preset confidence threshold, the operation instruction with the highest probability is selected as the output prediction instruction. When the confidence level is less than or equal to the preset confidence threshold, the prediction instruction set is discarded and a manual intervention request is triggered.

[0051] It should be noted that the preset confidence threshold is set based on the trust threshold in the neural decision-making mechanism, and the example value is 0.85.

[0052] The confidence score for the predicted instruction set is calculated using the following expression: ; In the formula, This indicates the confidence level of the predicted instruction set. Indicates from arrive definite integral, Denotes the Euclidean norm of a vector. Indicates the current time The probability distribution vector of the predicted instructions. This represents the time derivative of the predicted probability distribution. This represents the influence coefficient of the historical operation sequence. This represents the likelihood gradient operator. This represents a weighted historical operation sequence. This represents the likelihood gradient at a weighted historical operation sequence. This represents the exponentially decaying weight term. Indicates the time decay factor. This represents a time infinitesimal element.

[0053] It should be noted that the influence coefficient of historical operation sequence is determined by regression analysis of the matching degree between user historical operation data and real-time scene, and the example value is 0.7.

[0054] S3.5: Parse the semantic structure of the predicted instruction and generate an action descriptor containing the target position and action parameters; Specifically, the system receives the prediction instruction, applies instruction syntax parsing rules to decompose the semantic structure of the prediction instruction, extracts the target processor identifier from the prediction instruction, maps it to a three-dimensional spatial coordinate position, extracts the action type from the prediction instruction, converts it into a standard operation action code, extracts the parameter modifiers from the prediction instruction, quantifies them into a parameter set of action force value, angle value, and duration value, and combines the target processor's three-dimensional spatial coordinate position, standard operation action code, action force value, angle value, and duration value to generate an action descriptor.

[0055] S3.6: Based on the target location, construct a three-dimensional spatial model of the surface of the corresponding non-focal region processor; It should be noted that the process of constructing the surface three-dimensional space model of the corresponding non-focal region processor is as follows: access the historical non-focal region processor shell point cloud database, retrieve the processor point cloud dataset corresponding to the target location, read the user's eye coordinate position and three-dimensional viewing direction, calculate the origin position and coordinate axis direction of the user's eye coordinate system, align the depth axis with the three-dimensional viewing direction, make the horizontal plane and vertical plane perpendicular to the depth axis, apply rigid body transformation to transform the non-focal region processor shell point cloud database from the global coordinate system to the user's eye coordinate system, and output the transformed non-focal region processor shell point cloud database as the surface three-dimensional space model.

[0056] Specifically, the historical 3D point cloud database of processors in non-focus areas is accessed, and the processor shell point cloud dataset corresponding to the target location is retrieved. The processor shell point cloud dataset contains the 3D coordinates and normal vector information of the surface vertices. The coordinate transformation matrix is ​​applied to transform the processor shell point cloud dataset from the global coordinate system to the user's eye coordinate system. The origin of the eye coordinate system is the center position of the user's eyeball, which is aligned with the 3D line of sight vector, and the 3D spatial model of the surface is output.

[0057] S3.7: Calculate the ghost intensity field based on the motion parameters and generate the ghost intensity distribution field on the surface three-dimensional space model; Specifically, the motion force value parameter and motion direction vector parameter in the motion parameters are read, and the normalized scalar of the motion force value parameter is obtained as the intensity base coefficient. On the vertex coordinate set of the surface three-dimensional space model, the spatial distance value between each vertex and the target position is obtained to generate the basic intensity distribution field. The motion direction vector parameter and the vertex position vector are combined to generate the direction enhancement factor. The direction enhancement factor is superimposed on the basic intensity distribution field to output the ghost intensity distribution field.

[0058] Based on the motion parameters, the intensity field of the phantom is calculated as follows: ; In the formula, Represents the three-dimensional spatial coordinate vector of the current point. The value of the ghost image intensity field at that location, This represents the three-dimensional spatial coordinate vector of the current point. Indicates the strength reference coefficient. Describing the vector norm, Represents the action parameter vector. This indicates the operation of the natural exponent. Represents the three-dimensional spatial coordinate vector of the current point. The three-dimensional spatial coordinate vector of the target point The square of the Euclidean distance, Represents the three-dimensional spatial coordinate vector of the target point. Indicates the spatial attenuation coefficient. Indicates the direction modulation factor. This represents the spatial vector from the target point to the current point. This represents the unit vector indicating the direction of the action.

[0059] It should be noted that the intensity baseline coefficient is determined through retinal perception experiments to ensure the visibility of the ghost image, with an example value of 0.9; the spatial attenuation coefficient is calculated based on the human eye's visual field resolution model to control the range of ghost image diffusion, with an example value of 0.3; and the direction modulation factor is optimized based on the angle between the operation direction and the line of sight to enhance the guidance of the action, with an example value of 0.5.

[0060] S3.8: Real-time acquisition of user pupil diameter data, modulation of the transparency parameter of the ghost image intensity distribution field based on the user pupil diameter data, and generation of a ghost image field with modulated transparency; Specifically, the user's pupil diameter data is collected by an infrared pupil monitoring device, and the pupil diameter data is converted into a transparency scaling factor by applying the pupil diameter-ambient light response. The intensity floating-point value matrix of the ghost image intensity distribution field is read, and the intensity floating-point value matrix is ​​scaled using the transparency scaling factor to generate a transparency-modulated ghost image field.

[0061] S3.9: Convert the virtual image field after transparency modulation into a holographic phase map, and project a semi-transparent operation guide virtual image onto the surface of the physical device through a spatial light modulator.

[0062] Specifically, the intensity distribution data matrix of the virtual image field after transparency modulation is read, and the intensity distribution data matrix is ​​converted into complex wavefront distribution data by optical Fourier transform. The phase angle component of the complex wavefront distribution data is extracted to generate a holographic phase map. The holographic phase map is loaded into the phase control of the spatial light modulator. The spatial light modulator receives the coherent laser beam output by the laser generator and modulates the laser wavefront phase according to the phase control parameters. The modulated laser wavefront is diffracted by a freeform holographic grating to generate a semi-transparent operation guidance virtual image on the surface of the processor physical device in the non-focal region.

[0063] S4. Continuously capture the user's actual operation skeleton nodes through computer vision, and calculate the similarity between the node positions of the actual operation skeleton nodes and the semi-transparent operation guide phantom.

[0064] S4.1: Furthermore, by fusion of multiple sensors to capture the user's actual operation skeletal node data, a spatiotemporal sequence of skeletal motion is generated. Optical reconstruction is then performed on the semi-transparent operation-guided ghost image to generate a spatiotemporal sequence of ghost image skeletal motion. Specifically, the system captures three-dimensional coordinate data of user skeletal nodes using a depth camera, joint angular velocity data using inertial measurement, and electromyography (EMG) signal data using an EMG sensor. It then fuses the three-dimensional coordinate data, joint angular velocity data, and EMG signal data to generate a spatiotemporal sequence of skeletal motion. The system reads the semi-transparent operation guidance virtual light field projected onto the surface of the physical device, and applies an inverse light field diffraction algorithm to reconstruct the three-dimensional coordinate data of the virtual skeletal nodes. Finally, it fuses the reconstructed three-dimensional coordinate data with the virtual motion timestamp to generate a spatiotemporal sequence of virtual skeletal motion.

[0065] S4.2: Generate user motion intent feature data based on the gradient change characteristics of the spatiotemporal sequence of skeleton motion, and generate ghost motion intent feature data based on the gradient change characteristics of the spatiotemporal sequence of ghost skeleton motion. Specifically, the process involves obtaining the first-order time derivative of the spatiotemporal sequence of skeletal motion, extracting node motion velocity vectors, obtaining the second-order time derivative of the spatiotemporal sequence of skeletal motion, extracting node motion acceleration vectors, combining the node motion velocity vectors and node motion acceleration vectors to generate a four-dimensional motion dynamic feature vector, performing time decay weighting on the four-dimensional motion dynamic feature vector, and outputting user motion intent feature data. The same first-order and second-order time derivative operations are then performed on the spatiotemporal sequence of the virtual skeleton motion to generate a virtual motion dynamic feature vector, performing the same time decay weighting on the virtual motion dynamic feature vector, and outputting user motion intent feature data. Finally, the same first-order time derivative is performed on the spatiotemporal sequence of the virtual skeleton motion to output virtual motion intent feature data.

[0066] S4.3: Specifically, calculate the node position similarity between the user's motion intention feature data and the ghost image's motion intention feature data, expressed as follows: ; In the formula, Indicates the similarity of node positions. Indicates the weighting coefficient for the direction of motion. express The user's motion intention velocity vector at any given moment. express The velocity vector of the phantom's motion intention at any given moment. Represents the cosine similarity of velocity vectors. This represents the position weighting coefficient. Indicates Gaussian weights for positional deviation. express The positional deviation vector between the user's skeletal nodes and the phantom nodes at any given time. This represents the location tolerance coefficient. Indicates the time decay weight. This represents the time decay factor.

[0067] It should be noted that the motion direction weight coefficient is dynamically adjusted according to the operation type classification, with an example value of 0.6; the position weight coefficient is automatically calculated as the complement of the motion direction weight coefficient, with an example value of 0.4; the position tolerance coefficient is determined based on human motion accuracy experiments, with an example value of 50; and the time decay factor is calculated based on the visual short-term memory decay model, with an example value of 0.2.

[0068] S5. When the node position similarity exceeds the preset verification threshold, trigger the configuration change command corresponding to the prediction instruction to the marked non-focus area processor.

[0069] S5.1: Generate operational intent based on node position similarity, spatiotemporal sequence of skeletal motion, and spatiotemporal sequence of phantom skeletal motion. Figure 1 Coherence vector; Specifically, the process involves obtaining the dynamic time warping distance between the spatiotemporal sequences of skeletal motion and those of the virtual skeleton, extracting the motion trajectory curvature feature vectors from both the skeletal and virtual skeleton motion sequences, obtaining the cosine similarity between the two motion trajectory curvature feature vectors, and combining node position similarity, dynamic time warping distance, and cosine similarity to generate the operation. Figure 1 Consistency vector.

[0070] S5.2: Operational intention Figure 1 The consistency vector is used to perform spatiotemporal integration calculation and output dynamic confidence score. When the dynamic confidence score exceeds the preset verification threshold, an encrypted execution protocol packet is generated to encapsulate the configuration change command. It should be noted that the preset verification threshold is set based on the trust threshold of the neural decision-making mechanism, and the example value is 0.85.

[0071] Specifically, when the dynamic confidence score does not exceed the preset verification threshold, a retinal alert projection operation is triggered and command execution is suspended, awaiting manual review and confirmation or cancellation. When the dynamic confidence score exceeds the preset verification threshold, a configuration change command encapsulation operation is triggered. The configuration change command is read, the target processor identifier and parameter setting instructions are extracted, the configuration change command is encrypted using the target processor's public key, an encrypted configuration data block is generated, the user's digital signature private key is used to perform a hash signature on the encrypted configuration data block, digital signature data is generated, and the encrypted configuration data block, digital signature data, and timestamp are combined to generate an encrypted execution protocol packet.

[0072] Operational intention Figure 1 The consistency vector is used to perform spatiotemporal integration to output a dynamic confidence score, expressed as follows: ; In the formula, This indicates the dynamic confidence score. Represents the hyperbolic tangent function. Indicates meaning Figure 1 Consistent gain coefficient, Represents the spatial weight matrix. express Always in operation Figure 1 Consistency vector, Indicates the motion trajectory modulation factor. This represents the Sigmoid function. express The curvature vector of the skeleton's motion trajectory at any given time. Indicates the time decay weight. This represents the time decay factor.

[0073] It should be noted that, Figure 1The consistency gain coefficient is calibrated through human-computer interaction experiments to determine the relationship between intent intensity and confidence, with an example value of 1.2; the motion trajectory modulation factor optimizes the curvature influence weight based on the human joint motion limit model, with an example value of 0.8; the time decay factor is based on the visual short-term memory decay cycle, with an example value of 0.15.

[0074] S5.3: Send the encrypted execution protocol packet to the marked non-focus area processor.

[0075] Specifically, the encrypted execution protocol packet is encapsulated into an application layer data packet by the transport layer security protocol stack. The application layer data packet is then appended with the target non-focus area processor media, an access control address header, and an Internet Protocol address header to generate a network layer data frame. The network layer data frame is sent to the router device through the physical network interface. The router device performs routing table lookup and next-hop forwarding operations. The data frame is then transmitted through the switch link layer to the target non-focus area processor network interface.

[0076] S6. Real-time collection of new load data generated after the execution of configuration change commands, re-determination of whether the preset load threshold is exceeded, forming a dynamic monitoring and interactive closed loop.

[0077] S6.1: Collect new load data after the processor in the marked non-focus area executes the configuration change command, and recalculate the new comprehensive load index; Specifically, the CPU utilization percentage data is collected after the configuration change command is executed through the built-in performance counter of the non-focus area processor, the memory utilization percentage data is collected through the memory controller of the non-focus area processor, the network bandwidth utilization percentage data is collected through the network interface card of the non-focus area processor, and the core temperature data in degrees Celsius is collected through the thermal sensor of the non-focus area processor. The CPU utilization percentage data, memory utilization percentage data, network bandwidth utilization percentage data, and core temperature data in degrees Celsius are combined into new load data, and a new comprehensive load index is output based on the new load data.

[0078] S6.2: When the new comprehensive load index exceeds the preset load threshold, a pulsed red spectrum is projected onto the user's retina; when the new comprehensive load index does not exceed the preset load threshold, a steady-state green spectrum is projected. It should be noted that the preset load threshold is set based on the processor's safe operating temperature and performance saturation threshold; the example value is 0.85.

[0079] Specifically, the relationship between the new comprehensive load index and the preset load threshold is compared. When the new comprehensive load index exceeds the preset load threshold, the pulsed red spectral parameter set is selected. When the new comprehensive load index does not exceed the preset load threshold, the steady-state green spectral parameter set is selected. The spectral signal corresponding to the selected spectral parameter set is projected at a fixed coordinate position in the peripheral field of vision of the user's retina through the micro-projection optical engine.

[0080] S6.3: Track the user's iris movement trajectory. When the iris moves into the pulsed red spectrum region, generate a load migration command. When the iris gazes into the steady green spectrum region for more than a preset time, generate a configuration maintenance command. Specifically, the eye-tracking engine captures the user's iris center coordinate data sequence in real time, obtains the Euclidean distance value between the iris coordinates and the center coordinates of the pulsed red spectral region, and determines that the iris has moved to the pulsed red spectral region when the Euclidean distance value continuously decreases and is lower than the preset movement judgment threshold, and generates a load migration command. Simultaneously, it monitors the continuous overlap time value between the iris coordinates and the center coordinates of the steady green spectral region. When the overlap time value exceeds the preset duration threshold, it determines that the iris is staring at the steady green spectral region and generates a configuration maintenance command.

[0081] It should be noted that the preset motion detection threshold is set based on the minimum perceptible motion angle of the human eye, with an example value of 3; the preset duration threshold is set based on the shortest time required for visual confirmation, with an example value of 2000.

[0082] S6.4: After executing the load migration command or configuration maintenance instruction, new load data is collected again for judgment until the new comprehensive load index does not exceed the preset load threshold.

[0083] Specifically, after executing the load migration command or configuration maintenance instruction, a load data re-acquisition timer is started. When the timer triggers, new CPU utilization percentage data, new memory utilization percentage data, new network bandwidth utilization percentage data, and new core temperature data are collected from non-focus area processors. These data are then combined to generate new load data. Based on the new load data, a new comprehensive load index is output. The new comprehensive load index is compared with a preset load threshold. If the new comprehensive load index exceeds the preset load threshold, the load status spectral projection process is re-triggered; if the new comprehensive load index scalar value does not exceed the preset load threshold, the closed-loop monitoring process is terminated.

[0084] In summary, this invention projects a pulsed spectrum onto the user's peripheral retinal field of vision, using infrared spectroscopy to form a non-invasive light signal in the peripheral retinal field of vision. It leverages the high sensitivity of rod cells to dynamic light sources to transmit load alarms without interfering with the central focal processing of cone cells. The pulse frequency is linearly adjusted according to the load exceeding the threshold, allowing the cerebral cortex to determine the abnormality level through frequency analysis. Simultaneously, the spectral projection position is bound to the spatial coordinates of the target processor, providing a spatial reference for operational guidance.

[0085] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A visual interactive method for splicing processors, characterized in that: include, Real-time tracking of the user's iris movement trajectory, determining the processors within the current focal area of ​​the visual cone, and simultaneously marking processors in non-focal areas outside the focal area of ​​the visual cone; When the load data of the processor in the marked non-focal area exceeds a preset load threshold, a pulse spectrum is projected onto the user's peripheral visual field. Based on the pulse spectrum, a behavior prediction model is constructed by calling the user's historical operation sequence, and a prediction instruction is generated. A semi-transparent operation guidance ghost image is rendered on the physical device surface of the processor in the non-focal area. The system continuously captures the actual skeletal nodes of the user's operation using computer vision, and calculates the similarity between the node positions of the actual skeletal nodes and the semi-transparent operation guide phantom. When the node location similarity exceeds the preset verification threshold, the configuration change command corresponding to the prediction instruction is triggered and sent to the marked non-focus area processor. Real-time collection of new load data generated after the execution of configuration change commands, re-determination of whether the preset load threshold is exceeded, forming a dynamic monitoring and interactive closed loop.

2. The visual interaction method for the splicing processor as described in claim 1, characterized in that: The specific steps for simultaneously marking the non-focal region outside the focal region of the view frustum are as follows. The three-dimensional coordinate set of the iris reflection spot at the current moment is obtained by using a dual-band infrared ring light source, and the Euclidean displacement vector of the three-dimensional coordinate set of the iris reflection spot at the previous moment is calculated. Based on the predefined displacement-angle response mapping table, the Euclidean displacement vector is converted into the line-of-sight correction angle, and the original line-of-sight direction angle is calculated by combining the fixed light source-camera position vector of the device. The original line-of-sight angle is input into the head-mounted device's inertial measurement, and then transformed by the attitude rotation matrix to generate a three-dimensional line-of-sight vector in the world coordinate system; When the angle between the 3D view vector and the processor's spatial position vector is less than the preset focus threshold, the processor is determined to be within the current view frustum focus area; otherwise, it is marked as a non-focus processor.

3. The visual interaction method for the splicing processor as described in claim 2, characterized in that: The specific steps for projecting pulsed spectra onto the user's peripheral retinal field of vision are as follows. Collect load data of the processors in the marked non-focus areas and calculate the comprehensive load index of the processors in the non-focus areas; When the overall load index of the non-focus area processor exceeds the preset load threshold, the load feature vector is extracted from the load data, and the spatial coordinate position of the non-focus area processor and the user's eye coordinate position are obtained at the same time. Based on the three-dimensional gaze vector and the user's eye coordinate position, the mapping coordinates of the spatial coordinate position of the non-focal area processor in the peripheral visual field of the retina are obtained; Based on the load feature vector, pulse spectral parameters are generated, and the pulse spectrum corresponding to the pulse spectral parameters is projected at the mapping coordinates of the peripheral visual field of the retina.

4. The visual interaction method of the splicing processor as described in claim 3, characterized in that: The specific steps for generating the prediction instruction are as follows: Analyze the pulse spectrum to extract wavelength, frequency and light intensity data. Determine the processor load type in the non-focal area based on the wavelength data, determine the load urgency level based on the frequency data, and generate load characteristic identifiers. Retrieve the user's historical operation sequence, perform spatiotemporal weighted filtering of historical operation actions based on load characteristic identifiers, and output the weighted historical operation sequence; A behavior prediction model is constructed based on a weighted historical operation sequence. The hierarchical attenuation factor of the behavior prediction model is dynamically adjusted according to the light intensity data to generate a modulated behavior prediction model. The weighted historical operation sequence is input into the modulated behavior prediction model to generate a prediction instruction set. The confidence level of the prediction instruction set is calculated. When the confidence level exceeds the preset confidence threshold, the prediction instruction is output.

5. The visual interaction method for the splicing processor as described in claim 4, characterized in that: The process of rendering semi-transparent shadows involves the following steps: Parse the semantic structure of the predicted instruction and generate an action descriptor containing the target position and action parameters; Based on the target location, construct a three-dimensional spatial model of the surface of the corresponding non-focal region processor; Based on the motion parameters, the ghost intensity field is calculated, and the ghost intensity distribution field is generated on the surface three-dimensional space model; The system acquires real-time user pupil diameter data and modulates the transparency parameters of the ghost image intensity distribution field based on the user pupil diameter data to generate a transparency-modulated ghost image field. The virtual image field after transparency modulation is converted into a holographic phase map, and a semi-transparent operation guide virtual image is projected onto the surface of the physical device through a spatial light modulator.

6. The visual interaction method for the splicing processor as described in claim 5, characterized in that: The specific steps for calculating the similarity between the node positions of the semi-transparent guided phantom are as follows: By fusion of multiple sensors to capture the user's actual operation skeletal node data, a spatiotemporal sequence of skeletal motion is generated. Optical reconstruction is then performed on the semi-transparent operation-guided ghost image to generate a spatiotemporal sequence of ghost image skeletal motion. Based on the gradient change characteristics of the spatiotemporal sequence of skeletal motion, user motion intent feature data is generated; based on the gradient change characteristics of the spatiotemporal sequence of ghost skeleton motion, ghost motion intent feature data is generated. Calculate the node position similarity between the user's motion intention feature data and the ghost image's motion intention feature data.

7. The visual interaction method for the splicing processor as described in claim 6, characterized in that: The specific steps for sending the configuration change command to the marked non-focus area processor are as follows. Based on node position similarity, spatiotemporal sequence of skeleton motion, and spatiotemporal sequence of virtual skeleton motion, generate an operation intent consistency vector; Spatiotemporal integral calculation is performed on the consistency vector of operational intent to output a dynamic confidence score. When the dynamic confidence score exceeds a preset verification threshold, an encrypted execution protocol packet is generated to encapsulate the configuration change command. Send encrypted execution protocol packets to the marked non-focus area processor.

8. The visual interaction method of the splicing processor as described in claim 7, characterized in that: The specific steps for forming a dynamic monitoring and interactive closed loop are as follows. Collect new load data after the processor in the marked non-focus area executes the configuration change command, and recalculate the new comprehensive load index; When the new comprehensive load index exceeds the preset load threshold, a pulsed red spectrum is projected onto the user's retina; when the new comprehensive load index does not exceed the preset load threshold, a steady-state green spectrum is projected. Track the user's iris movement trajectory. When the iris moves into the pulsed red spectrum region, generate a load migration command. When the iris gazes into the steady green spectrum region for more than a preset time, generate a configuration maintenance command. After executing the load migration command or configuration maintenance instruction, new load data is collected again for judgment until the comprehensive load index does not exceed the preset load threshold.

Images