Method and device for interacting with a large-format painting based on infrared motion capture

Abstract

The application discloses an infrared motion capture-based large painting interaction method and device, and relates to the technical field of human-computer interaction.The method comprises the following steps: based on an ultrahigh-definition large display terminal and an infrared motion capture system, collecting user skeletal joint data, and generating a screen interaction cursor coordinate sequence; recognizing the motion characteristics of a gesture action, determining an interaction intention, and generating an operation instruction; when a focusing instruction is recognized, intercepting image data to be segmented; calling an improved cuckoo catfish optimization algorithm to optimize the optimal threshold parameter set of image segmentation; using an image segmentation model to disassemble the image data to be segmented into brush stroke layer data, ink color layer data, and seal layer data; and fusing and rendering the brush stroke layer data, the ink color layer data, and the seal layer data with preset painting analysis information, and performing layered and superimposed display.The application solves the problems of low interaction precision, single intention recognition, and poor local detail layering and analysis capability of large paintings in the prior art.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a method and device for interacting with large-scale paintings based on infrared motion capture. Background Technology

[0002] With the rapid development of ultra-high-definition display technology, ultra-high-definition large-format display terminals are increasingly widely used in cultural exhibition venues such as museums and art galleries. In the digital display of large-format paintings such as traditional ink paintings, viewers often can only passively observe, lacking a deep interactive experience.

[0003] In existing technologies, human-computer interaction for large-scale paintings typically relies on touchscreens or handheld devices. However, touchscreens are extremely expensive on large-scale devices and struggle to enable multi-user, long-distance collaborative interaction, while handheld devices disrupt the immersive viewing experience. Although infrared motion capture technology can achieve contactless interaction, it suffers from the following drawbacks in practical applications: 1) Ray calculation based on hand gestures is easily affected by human skeletal vibrations, leading to inaccurate cursor positioning; 2) The interaction intent recognition is relatively simple, making it difficult to achieve smooth, stepless scaling and precise area focusing; 3) When users focus on a specific detail of a painting, the system lacks the ability to intelligently deconstruct and layer the details of the painting (such as brushstrokes, ink density, and seals), which fails to meet the needs of professional-level art appreciation. 4) Traditional image segmentation algorithms are extremely sensitive to thresholds when extracting elements from traditional Chinese ink paintings, and are prone to getting stuck in local optima, resulting in broken brushstrokes, mixed ink colors, or incomplete extraction of seals. Summary of the Invention

[0004] This invention provides a method and device for interactive large-scale paintings based on infrared motion capture. This invention solves the problems of low accuracy, limited intent recognition, and poor ability to analyze local details in existing technologies for large-scale painting interaction.

[0005] In a first aspect, embodiments of the present invention provide an interactive method for large-scale paintings based on infrared motion capture, the method comprising: Based on an ultra-high-definition large-format display terminal and an infrared motion capture system, the system collects user skeletal joint data, calculates the intersection of the hand pointing ray and the screen, and generates a sequence of screen interactive cursor coordinates. Based on the screen interaction cursor coordinate sequence, the motion characteristics of the gesture are identified, the interaction intent is determined, and the corresponding operation command is generated. The screen is continuously zoomed and panned according to the operation instructions. When the focus instruction is detected, the image of the currently locked area of ​​the cursor is captured as the image data to be segmented. The improved cuckoo-catfish optimization algorithm is invoked, and the population is initialized by chaotic sequence, a convergence factor is introduced, and the PSO mechanism is fused for iterative optimization, outputting the optimal threshold parameter set for image segmentation. The optimal threshold parameter set is used to drive a pre-built image segmentation model to decompose the image data to be segmented into brush stroke layer data, ink layer data and seal layer data. The brushstroke layer data, ink layer data, and seal layer data are fused and rendered with preset painting analysis information, and then displayed in layers and overlays on an ultra-high-definition giant display terminal.

[0006] The technical solution provided in this application has at least the following beneficial effects: By acquiring skeletal key points through an infrared motion capture system and constructing a pointing ray from the elbow to the wrist joint and projecting it onto the screen, the system effectively filters out minute natural hand tremors, improving the stability and accuracy of cursor positioning on the large screen. Combined with gesture, displacement, and depth feature analysis, it achieves seamless, smooth interaction for translation, scaling, and focusing. Specifically designed for ink wash paintings, an improved cuckoo-catfish optimization algorithm is innovatively proposed to find the optimal segmentation threshold. Logistic chaotic mapping is used to initialize the population to ensure global search capability, and Levy flight (strong exploration) is employed before, during, and in the later stages of iteration. The PSO (Planetary Optimization and Search) fusion strategy, combining catfish-based perturbation defense (balanced development and exploration) and PSO fine-grained search (strong local development), completely solves the problem of traditional threshold segmentation easily getting trapped in local extrema, leading to broken brushstrokes or incorrect ink extraction. This improves image segmentation accuracy and anti-interference capabilities. It transforms complex computer vision segmentation results into intuitive artistic analysis and display. By extracting three independent layers of data—brushstrokes, ink color, and seals—and fusing them with metadata (such as interpretations and color spectra), it achieves an immersive digital interaction of large-scale paintings, moving from "overall distant view" to "local deconstruction," greatly enhancing the value of cultural dissemination and art exhibitions.

[0007] In one optional implementation, the user skeletal joint data includes a sequence of coordinates of key skeletal points in the upper limb and a sequence of coordinates of skeletal joints in the hand. The coordinate sequence of key skeletal points of the upper limb includes the world coordinates of the shoulder joint, elbow joint, and wrist joint at each moment. The hand bone joint coordinate sequence includes the wrist joint coordinates and the set of finger joint coordinates for each finger at each moment.

[0008] In one alternative implementation, based on an ultra-high-definition large-format display terminal and an infrared motion capture system, user skeletal joint point data is collected, the intersection of the hand pointing ray and the screen is calculated, and a screen interaction cursor coordinate sequence is generated, including: An infrared motion capture system is used to collect data on the user's skeletal joint points. Using the line connecting the elbow to the wrist as the direction of the interactive ray, and based on the world coordinates of the elbow and wrist joints in the upper limb key bone point coordinate sequence, a hand pointing ray is constructed from the elbow to the wrist, as shown in the formula: In the formula, The hand points to the ray; For elbow joint world coordinates and wrist joint world coordinates; The intersection of the pointing ray of the hand and the plane equation of the ultra-high-definition giant display terminal is found. Using the camera calibration matrix, the three-dimensional intersection point is projected onto the two-dimensional coordinate system of the screen to obtain the screen cursor coordinates. The formula is as follows: In the formula, These are the screen cursor coordinates; This represents the depth value along the Z-axis in both elbow and wrist world coordinates. For ray parameters; Calibrate the camera matrix; Set a timestamp for each screen cursor coordinate to obtain a sequence of screen interactive cursor coordinates that change over time.

[0009] In one optional implementation, based on the screen interaction cursor coordinate sequence, the motion characteristics of the gesture are identified, the interaction intent is determined, and a corresponding operation command is generated, including: Based on the screen interaction cursor coordinate sequence, motion features of the gesture actions are extracted. These motion features include the cursor displacement vector and the hand depth change, as shown in the formula: In the formula, The cursor displacement vector; For the time in the screen interaction cursor coordinate sequence l , l -1 is the screen cursor coordinate; This represents the change in hand depth. For a moment l , l -1 represents the depth value of the elbow joint in the world coordinate system along the Z-axis. Based on the hand bone joint coordinate sequence and the motion features, the interaction intent is determined and the corresponding operation command is generated. The interaction intent includes translation intent, zoom intent and focus intent, and the corresponding operation command includes translation command, zoom command and focus command.

[0010] In one optional implementation, based on the hand bone joint coordinate sequence and the motion features, the interaction intent is determined and a corresponding operation command is generated, including: The hand bone joint coordinate sequence is input into a pre-trained gesture recognition model to obtain the corresponding gesture recognition results. The gesture recognition results include fist gestures, open gestures, and push gestures. The gesture recognition model is constructed based on the LSTM algorithm. If the gesture recognition result is a clenched fist gesture, and the absolute value of the cursor displacement vector in the motion features is greater than the displacement threshold, then the interaction intent is determined to be a translation intent, and the corresponding operation instruction is a translation instruction. If the gesture recognition result is an open gesture, and the rate of change of the distance between the hands is greater than the rate of change threshold, then the interaction intent is determined to be a scaling intent, and the corresponding operation instruction is a scaling instruction. If the gesture recognition result is a pushing gesture, and the change in hand depth in the motion feature is greater than the change threshold within a preset continuous time window, and the cursor displacement vector of the screen cursor coordinates within a preset displacement tolerance radius centered on the initial screen cursor coordinates within the time window does not exceed the area, then the interaction intent is determined to be a focus intent, and the corresponding operation instruction is generated as a focus instruction.

[0011] In one optional implementation, the screen is continuously zoomed and panned according to the operation command. When a focus command is detected, the image of the currently locked cursor area is captured as the image data to be segmented, including: If the operation command is a panning command or a zoom command, then the ultra-high-definition large-format display terminal will perform stepless zooming and panning of the screen according to the operation command. If the operation command is a focus command, then the specific area of ​​the giant painting on the ultra-high-definition giant display terminal with the screen cursor coordinates is taken as the current cursor lock area, and the screen cursor coordinates are taken as the center coordinates. Based on the center coordinates, the positioning rectangle area of ​​the ultra-high-definition giant display terminal is determined, and based on the positioning rectangle area, the image of the currently cursor-locked area is extracted as the image data to be segmented.

[0012] In one alternative implementation, an improved cuckoo-catfish optimization algorithm is invoked. This algorithm initializes the population using a chaotic sequence, introduces a convergence factor, and incorporates a PSO mechanism for iterative optimization, outputting the optimal threshold parameter set for image segmentation, including... The segmentation threshold parameter vector of the image data to be segmented is encoded as the position vector of an individual in the improved cuckoo catfish optimization algorithm. A fitness function is set, and historical image data is used as a simulation environment. The segmentation threshold parameter vector includes brush stroke segmentation threshold, ink color segmentation threshold, and seal segmentation threshold. The chaotic sequence is generated using the Logistic mapping, and then mapped to the solution space of individuals in the improved cuckoo-catfish optimization algorithm to obtain the initial population. Using the fitness function, calculate the fitness value of each initial individual in the initial population, and determine the optimal position of each initial individual and the global optimal individual in the initial population based on the fitness value. In the early stages of iteration, a Levy flight mechanism is introduced to perform position updates of cuckoo nest parasitism on the initial population or the population updated in the last iteration, resulting in an updated population. In the middle of the iteration, based on the individual optimal position and the global optimal individual, the PSO mechanism and random perturbation are introduced to perform position updates of the catfish defense behavior on the population updated in the last update, and the updated population is obtained. In the later stages of the iteration, based on the individual optimal position and the global optimal individual, the population updated in the previous update is subjected to a fine-grained search behavior dominated by the PSO mechanism to obtain the updated population. Using the fitness function, calculate the fitness value of each updated individual in the updated population, and update the global best individual based on the fitness value; Repeatedly update the position of the population. When the number of iterations reaches the maximum number of iterations or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the globally optimal individual. The position vector of the globally optimal individual is decoded to obtain the optimal threshold parameter set for image segmentation.

[0013] In one optional implementation, the optimal threshold parameter set is used to drive a pre-built image segmentation model to decompose the image data to be segmented into brushstroke layer data, ink layer data, and stamp layer data, including: The optimal threshold parameter set is used to drive a pre-built image segmentation model, which is built based on an adaptive threshold segmentation algorithm with multi-feature fusion, including a feature preprocessing layer, a parameter mapping layer, a threshold segmentation execution layer, and a post-processing optimization layer. The feature preprocessing layer is used to preprocess the image data to be segmented to obtain the corresponding gradient feature map, grayscale feature map and hue saturation feature map. The feature preprocessing layer includes a gradient calculation module, a grayscale conversion module, and an HSV color space conversion module; The gradient calculation submodule calculates the image edge intensity based on the Sobel or Canny operator, and is used to perform gradient calculation on the image data to be segmented, generate gradient feature maps, and capture high-frequency brushstroke information of the painting. The grayscale conversion module is used to convert the image data to be segmented in RGB format into a single-channel grayscale image, generate a grayscale feature map, and analyze the tonal variations of the ink painting. The HSV color space conversion module is used to convert the image data to be segmented in RGB format to HSV space, extract the hue component and saturation component, generate a hue and saturation feature map, and identify specific color (such as red stamp) regions. The optimal threshold parameter set is input to the parameter mapping layer, and the optimal threshold parameter set is mapped to a specific segmentation decision threshold set that the threshold segmentation execution layer can recognize; The specific segmentation decision threshold set includes a brushstroke segmentation decision threshold, a multi-level ink color segmentation boundary, and a stamp segmentation lower limit threshold. The parameter mapping layer maps the optimal brushstroke segmentation threshold to a gradient binarized brushstroke segmentation decision threshold, maps the ink color segmentation threshold to a multi-level ink color segmentation boundary of a grayscale histogram, and maps the stamp segmentation threshold to a saturation-filtered stamp segmentation lower limit threshold, and locks the target by combining it with a preset red hue range. Based on the specific segmentation decision threshold set, a threshold segmentation execution layer is used to perform threshold segmentation on gradient feature map, grayscale feature map and hue saturation feature map to obtain the initial brush stroke bone layer, initial ink layer and initial seal layer. The threshold segmentation execution layer uses a hard threshold binarization algorithm to select pixels in the gradient feature map whose gradient magnitude is greater than the multi-level ink color segmentation boundary to generate the initial brush stroke bone layer. The multi-level threshold quantization algorithm is used to divide the grayscale feature map into regions according to the multi-level ink color segmentation boundary to obtain regions that distinguish the background white space and ink color regions of different shades, generating the initial ink color layer. The color range masking algorithm is used to extract pixel regions in the hue saturation feature map that simultaneously satisfy the red hue and saturation greater than or equal to the lower limit threshold of the stamp segmentation in the HSV feature space, obtaining the initial stamp layer. Using a post-processing optimization layer, connected component analysis and hole filling are performed on the initial brush stroke layer, initial ink layer, and initial stamp layer to remove isolated interference pixels and obtain independent brush stroke layer data, ink layer data, and stamp layer data.

[0014] In one optional implementation, brushstroke layer data, ink layer data, and stamp layer data are fused and rendered with preset painting analysis information, and then displayed in layers and overlays on an ultra-high-definition large-format display terminal, including: A database of artwork analysis information is pre-built to store metadata associated with the large-scale artworks on the ultra-high-definition large-scale display terminal. The metadata includes text descriptions of brushstroke styles, color gradations of ink, and seal interpretation information. Using the graphics rendering engine of the ultra-high-definition large-format display terminal, multi-layer color mapping and texture generation are performed on the brush stroke layer data, ink layer data and seal layer data to generate corresponding brush stroke view, ink color view and seal view. Brush stroke layer rendering: Apply a stroke rendering algorithm to the brush stroke layer data to map areas with a pixel value of 1 to a high-contrast line color (such as dark brown) to generate a brush stroke visual. Ink layer rendering: Based on the levels divided by the ink color segmentation threshold, different areas of the ink color layer data are filled with the corresponding ink color spectrum in the database to restore the ink wash effect and generate an ink color visual image. Seal layer rendering: Fill the seal layer data with vermilion red (RGB: 200, 30, 20) and overlay a semi-transparent shadow effect to enhance the three-dimensionality, generating a seal visual image; Based on the focus command, the pen stroke view, ink color view and seal view are displayed in turn in layers until the superposition cycle is reached, and then the next step is entered. Using a connected component analysis algorithm, the geometric center coordinates of the brushstroke view, ink color view, and seal view are calculated, and the corresponding painting analysis information is called from the painting analysis information database to generate a floating UI control with the anchor point coordinates as the geometric center coordinates. Based on the focus command, a weighted fusion rendering is performed on the pen stroke view, ink color view, and seal view to obtain a rendered view. This rendered view, with floating UI controls, is then displayed on an ultra-high-definition large-format display terminal. The formula is as follows: In the formula, To render the view; The image data to be segmented; These include brushstroke visualizations, ink color visualizations, and seal visualizations. For fusion weighting coefficients.

[0015] Secondly, embodiments of the present invention provide a large-scale painting interaction device based on infrared motion capture, used to implement a large-scale painting interaction method based on infrared motion capture, the device comprising: The infrared motion capture unit is used to collect user skeletal joint data, calculate the intersection of the hand pointing ray and the screen, and generate a screen interactive cursor coordinate sequence based on an ultra-high-definition large-format display terminal and an infrared motion capture system. An interaction intent recognition unit is used to identify the motion characteristics of gestures based on the screen interaction cursor coordinate sequence, determine the interaction intent, and generate corresponding operation instructions. The interactive region locking unit is used to perform stepless zooming and panning of the screen according to the operation command. When the focus command is detected, the image of the current cursor locked area is captured as the image data to be segmented. The segmentation threshold optimization unit is used to call the improved cuckoo-catfish optimization algorithm, which initializes the population through chaotic sequences, introduces convergence factors, and integrates the PSO mechanism to perform iterative optimization and outputs the optimal threshold parameter set for image segmentation. The image segmentation unit is used to drive a pre-built image segmentation model using the optimal threshold parameter set to decompose the image data to be segmented into brush stroke layer data, ink layer data and seal layer data. The fusion rendering unit is used to fuse and render brush stroke layer data, ink layer data, and seal layer data with preset painting analysis information, and then display them in layers and overlays on an ultra-high-definition large-format display terminal.

[0016] A third aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of a large-scale painting interaction method based on infrared motion capture provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the functional units of a large-scale painting interactive device based on infrared motion capture provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] The present invention will be further described below with reference to the accompanying drawings.

[0021] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0022] like Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0023] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0024] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for a giant painting interactive device based on infrared motion capture.

[0025] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the infrared motion capture-based giant painting interaction device stored in the memory 1005 through the processor 1001, and executes the infrared motion capture-based giant painting interaction method provided in the embodiment of the present invention.

[0026] Reference Figure 2 The present invention provides an interactive method for large-scale paintings based on infrared motion capture, the method comprising: S201: Based on an ultra-high-definition large-format display terminal and an infrared motion capture system, it collects user skeletal joint data, calculates the intersection of the hand pointing ray and the screen, and generates a sequence of screen interactive cursor coordinates. S202: Based on the screen interaction cursor coordinate sequence, identify the motion characteristics of the gesture, determine the interaction intention, and generate the corresponding operation command; S203: Perform stepless zooming and panning of the screen according to the operation command. When the focus command is detected, the image of the currently locked area of ​​the cursor is captured as the image data to be segmented. S204: Invoke the improved cuckoo-catfish optimization algorithm, initialize the population through chaotic sequence, introduce a convergence factor, and integrate the particle swarm optimization (PSO) mechanism to perform iterative optimization and output the optimal threshold parameter set for image segmentation; S205: Using the optimal threshold parameter set to drive the pre-built image segmentation model, the image data to be segmented is decomposed into brush stroke layer data, ink layer data and seal layer data. S206: The brush stroke layer data, ink layer data, and seal layer data are fused and rendered with the preset painting analysis information, and then displayed in layers and overlays on an ultra-high-definition large-format display terminal.

[0027] The technical solution provided in this application has at least the following beneficial effects: By acquiring skeletal key points through an infrared motion capture system and constructing a pointing ray from the elbow to the wrist joint and projecting it onto the screen, the system effectively filters out minute natural hand tremors, improving the stability and accuracy of cursor positioning on the large screen. Combined with gesture, displacement, and depth feature analysis, it achieves seamless, smooth interaction for translation, scaling, and focusing. Specifically designed for ink wash paintings, an improved cuckoo-catfish optimization algorithm is innovatively proposed to find the optimal segmentation threshold. Logistic chaotic mapping is used to initialize the population to ensure global search capability, and Levy flight (strong exploration) is employed before, during, and in the later stages of iteration. The PSO (Planetary Optimization and Search) fusion strategy, combining catfish-based perturbation defense (balanced development and exploration) and PSO fine-grained search (strong local development), completely solves the problem of traditional threshold segmentation easily getting trapped in local extrema, leading to broken brushstrokes or incorrect ink extraction. This improves image segmentation accuracy and anti-interference capabilities. It transforms complex computer vision segmentation results into intuitive artistic analysis and display. By extracting three independent layers of data—brushstrokes, ink color, and seals—and fusing them with metadata (such as interpretations and color spectra), it achieves an immersive digital interaction of large-scale paintings, moving from "overall distant view" to "local deconstruction," greatly enhancing the value of cultural dissemination and art exhibitions.

[0028] In one optional implementation, the user skeletal joint data includes a sequence of coordinates of key skeletal points in the upper limb and a sequence of coordinates of skeletal joints in the hand. The coordinate sequence of key skeletal points of the upper limb includes the world coordinates of the shoulder joint, elbow joint, and wrist joint at each moment. The hand bone joint coordinate sequence includes the wrist joint coordinates and the set of finger joint coordinates for each finger at each moment.

[0029] In one alternative implementation, based on an ultra-high-definition large-format display terminal and an infrared motion capture system, user skeletal joint point data is collected, the intersection of the hand pointing ray and the screen is calculated, and a screen interaction cursor coordinate sequence is generated, including: S2011: Uses an infrared motion capture system to collect user skeletal joint data; In this embodiment, the infrared motion capture system typically consists of multiple infrared cameras. It acquires three-dimensional coordinates through reflective markers. Due to the slight noise in infrared cameras, the acquired raw world coordinates need to be smoothed by a Kalman filter (or moving average filter). For the upper limb, the acromion, olecranon of the elbow, and styloid process of the ulna / radial bone are selected as key points. For the hand, the wrist joint and the metacarpophalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints of the index finger to the little finger are selected as the set of coordinates of the hand skeletal joints. S2012: Using the line connecting the elbow to the wrist as the direction of the interactive ray, and based on the world coordinates of the elbow and wrist joints in the upper limb key bone point coordinate sequence, construct a hand pointing ray from the elbow to the wrist, using the following formula: In the formula, The hand points to the ray; For elbow joint world coordinates and wrist joint world coordinates; In this embodiment, traditional interaction often uses the index finger to point, but when a person points in mid-air, the index finger is prone to slight multi-degree-of-freedom tremors. This embodiment innovatively uses the "elbow-wrist line" as a stable baseline. In the calculation, it does not simply take the subtraction of two points, but treats the upper arm and forearm as rigid body links, uses the shoulder joint coordinates as an auxiliary reference, and eliminates the relative offset of the elbow caused by the swaying of the torso, thereby ensuring that the ray direction is absolutely pointing near the screen normal direction, which greatly improves the stability of long-distance pointing. S2013: Find the intersection of the hand pointing ray and the plane equation of the ultra-high-definition giant display terminal. Using the camera calibration matrix, project the three-dimensional intersection point onto the two-dimensional coordinate system of the screen to obtain the screen cursor coordinates. The formula is: In the formula, These are the screen cursor coordinates; This represents the depth value along the Z-axis in both elbow and wrist world coordinates. For ray parameters; Calibrate the camera matrix; In this embodiment, ultra-high-definition giant display terminals usually have a certain physical curvature or installation tilt angle, so the Z-axis depth cannot be simply cut off directly. In this step, the real three-dimensional spatial plane equation of the screen plane is obtained in advance through the calibration board. The ray equation and the plane equation are solved simultaneously to obtain the three-dimensional intersection point. Then, it is multiplied by the camera calibration matrix (usually a 3x4 projection matrix) that includes lens distortion correction parameters, and accurately mapped to the pixel coordinate system of the screen. The calculation of the ray parameters ensures that the intersection point is always located between the viewer and the screen, eliminating the "phantom intersection point" caused by the ray extending behind the screen. S2014: Set a timestamp for each screen cursor coordinate to obtain a sequence of screen interactive cursor coordinates that change over time.

[0030] In one optional implementation, based on the screen interaction cursor coordinate sequence, the motion characteristics of the gesture are identified, the interaction intent is determined, and a corresponding operation command is generated, including: S2021: Based on the screen interaction cursor coordinate sequence, extract the motion features of the gesture actions. The motion features include the cursor displacement vector and the hand depth change, as shown in the formula: In the formula, The cursor displacement vector; For the time in the screen interaction cursor coordinate sequence l , l -1 is the screen cursor coordinate; This represents the change in hand depth. For a moment l , l -1 represents the depth value of the elbow joint in the world coordinate system along the Z-axis. In this embodiment, the cursor displacement vector only reflects the velocity in the two-dimensional plane. Further calculation of its second-order difference (acceleration) based on the first-order difference is used to determine whether the gesture is "uniform sliding" or "sudden stop". The extraction of the hand depth change is particularly critical. Since the Z-axis (depth) accuracy of the infrared camera is usually lower than that of the XY axis, the median Z-axis of multiple consecutive frames (such as 5 frames) is used here to filter out depth noise. S2022: Based on the hand bone joint coordinate sequence and the motion features, determine the interaction intent and generate corresponding operation instructions. The interaction intent includes translation intent, zoom intent and focus intent, and the corresponding operation instructions include translation instructions, zoom instructions and focus instructions. In this embodiment, a single displacement or gesture cannot constitute a safe and reliable command. An AND logic gate of "static gesture (morphology) + dynamic feature (kinematics)" is used for determination.

[0031] In one optional implementation, based on the hand bone joint coordinate sequence and the motion features, the interaction intent is determined and a corresponding operation command is generated, including: S20221: Input the hand bone joint coordinate sequence into a pre-trained gesture recognition model to obtain the corresponding gesture recognition results. The gesture recognition results include fist gestures, open gestures, and push gestures. The gesture recognition model is constructed based on the Long Short-Term Memory (LSTM) algorithm. S20222: If the gesture recognition result is a clenched fist gesture, and the absolute value of the cursor displacement vector in the motion features is greater than the displacement threshold, then the interaction intention is determined to be a translation intention, and the corresponding operation instruction is a translation instruction. S20223: If the gesture recognition result is an open gesture, and the rate of change of the distance between the hands is greater than the rate of change threshold, then the interaction intent is determined to be a scaling intent, and the corresponding operation instruction is a scaling instruction. S20224: If the gesture recognition result is a pushing gesture, and the change in hand depth in the motion feature is greater than the change threshold within a preset continuous time window, and the cursor displacement vector of the screen cursor coordinates within the preset displacement tolerance radius centered on the initial screen cursor coordinates within the time window does not exceed the area, then the interaction intent is determined to be a focusing intent, and the corresponding operation instruction is generated as a focusing instruction.

[0032] In one optional implementation, the screen is continuously zoomed and panned according to the operation command. When a focus command is detected, the image of the currently locked cursor area is captured as the image data to be segmented, including: S2031: If the operation command is a panning command or a zoom command, then the ultra-high-definition large-format display terminal's screen will be continuously zoomed and panned according to the operation command. In this embodiment, the operation command does not directly modify the UV coordinates of the texture, but is passed into the camera model of the graphics rendering engine (such as the underlying engine based on OpenGL / Vulkan). The camera's field of view is modified in real time to achieve stepless scaling, and the LookAt parameter is modified to achieve translation. To ensure that the screen is not torn, vertical synchronization technology is used, and a double buffering mechanism is added before and after each frame rendering. S2032: If the operation command is a focus command, then the specific area of ​​the giant painting on the ultra-high-definition giant display terminal with the screen cursor coordinates is taken as the current cursor locking area, and the screen cursor coordinates are taken as the center coordinates. In this embodiment, when a focus command is received, the timestamp of the trigger moment and the cursor coordinates are recorded. Considering the inertia when the gesture sinks, tens of milliseconds are backdated, and the coordinates when the cursor speed is close to 0 are taken as the true center coordinates to improve the locking accuracy. S2033: Based on the center coordinates, determine the positioning rectangle area of ​​the ultra-high-definition giant display terminal, and based on the positioning rectangle area, extract the image of the currently cursor-locked area as the image data to be segmented; In this embodiment, since ultra-high-definition images (such as 8K) are usually stored in video memory as compressed textures (such as ASTC / BC7), direct cropping would result in mosaic-like blocks at the edges. In this step, the GPU shader dynamically calculates a rectangular area with an aspect ratio adapted to the screen, centered on the center coordinates and based on the current scaling factor. Uncompressed RGBA pixel-level readback is performed through the GPU's Gl.readPixels or a hardware-level frame buffer object to ensure that the image data to be segmented sent to the subsequent image segmentation module maintains absolute original image quality.

[0033] In one alternative implementation, an improved cuckoo-catfish optimization algorithm is invoked. This algorithm initializes the population using a chaotic sequence, introduces a convergence factor, and incorporates a PSO mechanism for iterative optimization, outputting the optimal threshold parameter set for image segmentation, including... S2041: Encode the segmentation threshold parameter vector of the image data to be segmented into the position vector of an individual in the improved cuckoo catfish optimization algorithm, set the fitness function, and use historical image data as a simulation environment. The segmentation threshold parameter vector includes brush stroke segmentation threshold, ink color segmentation threshold, and seal segmentation threshold. S2042: Use Logistic mapping to generate chaotic sequences, and map the chaotic sequences to the solution space of individuals in the improved cuckoo-catfish optimization algorithm to obtain the initial population; The formula is: In the formula, For the first n+ 1. n There are several chaotic variables whose values ​​range from [0, 1]. The stability coefficient is typically 4. This sequence is ergodic and random, ensuring that the initial population is uniformly distributed in the solution space, avoiding getting trapped in local optima, which is superior to traditional random initialization. n Indicator of chaotic variables; In the formula, For the initial population, the first i An initial individual; For the first i One chaotic variable; To determine the upper and lower bounds of the solution space; i For individual indicators; t This represents the current iteration number; In the formula, For the initial population, the first i The initial velocity of each initial individual; For interval A function for randomly selecting values; S2043: Using the fitness function, calculate the fitness value of each initial individual in the initial population, and determine the optimal position of each initial individual and the global optimal individual in the initial population based on the fitness value. S2044: In the early stages of the iteration, the Levy flight mechanism is introduced to perform position updates of cuckoo nest parasitism behavior on the initial population or the population updated in the last iteration, resulting in an updated population. exist t < hour, This represents the maximum number of iterations, and we are in the early stages of iteration, generating random numbers for exploration. ,like , Let the host discovery probability be 0.25. Then, the location is updated via Levy flight using the following formula: In the formula, For the first t+ 1st iteration i A newer individual; For the first t+ 1st iteration i Each updated individual, in the initial iteration, For the initial individual; This is the step size control factor; This is the symbol for point-to-point multiplication; For Levy flight items; like This means that the current solution (nest) is of poor quality or trapped in a local extremum, and the current search region must be abandoned to reconstruct a new solution, as shown in the formula: In the formula, For the first t+ 1st iteration i A newer individual; A random number in the interval [0,1]. In this embodiment, this mechanism enables the search individual to conduct a fine search within a small range with a high probability, and to cross to the other end of the threshold space with a low probability (e.g., suddenly jumping from the ink threshold range to the stamp threshold range). Combined with the host's probability-based mechanism of discarding inferior nests, it can quickly lock in potential threshold ranges. S2045: In the middle of the iteration, based on the individual optimal position and the global optimal individual, the PSO mechanism and random perturbation are introduced to perform position updates of the catfish defense behavior on the population updated in the last update, and the updated population is obtained. exist At this stage, we are in the middle of the iteration. The focus is on balancing development and exploration, implementing the fusion of catfish defense and PSO behavior, and enhancing random perturbations. The formula is as follows: In the formula, For the first t+ 1st iteration i The rate at which each updated individual is updated; For the first t The iteration of the ... i The update speed of each updated individual, in the initial iteration, The initial velocity; For the first t The iteration of the ... i The best historical position; Accelerate one's own cognition; The acceleration coefficient of social cognition; Let be a random perturbation vector in the interval [0,1]. For the first t The globally optimal individual in the next iteration; This is the inertia weighting coefficient; The formula is: In the formula, For the first t+ 1st iteration i A newer individual; The mid-term defensive perturbation coefficient, with a value range of [0.1, 0.5], simulates the defensive random walk of the catfish; The position of a random individual in the population is introduced to prevent the population from prematurely clustering in local optima in the medium term. For the first t The convergence factor of the next iteration; In the formula, These are the maximum and minimum values ​​of the convergence factor; This represents the maximum number of iterations. In this embodiment, the introduced "mid-term defense perturbation coefficient" (set to 0.3) is equivalent to superimposing a factor related to the "random individual" on the gravitational field where the PSO approaches its own optimum and the global optimum. "The related repulsive field, if the current optimum is a local extremum, this perturbation can 'pull' some individuals out, and at the same time, the convergence factor..." from (e.g., 0.9) towards (e.g., 0.4) linearly decreases, controlling the initial stage to retain a large inertia (to continue exploring), and the later stage to reduce the inertia (to prepare for convergence); S2046: In the later stages of the iteration, based on the individual's optimal position and the global optimal individual, the population updated in the previous update is updated by performing a fine-grained search behavior dominated by the PSO mechanism to obtain the updated population. exist At this stage, which is in the later stages of iteration, the focus is on convergence towards the global optimum, performing a fine-grained search behavior dominated by PSO, as shown in the formula: In the formula, For the first t+ 1st iteration i A newer individual; In this embodiment, the catfish disturbance is removed and the standard PSO formula is used completely. At this time, the inertial weight is very small. The individual is like performing a grid scan near the local optimum, fine-tuning the boundary, so that the "dark, light, clear" boundary of the ink reaches pixel-level precision. S2047: Using the fitness function, calculate the fitness value of each updated individual in the updated population, and update the global best individual based on the fitness value; S2048: Repeatedly update the position of the population. When the number of iterations reaches the maximum number of iterations or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the globally optimal individual. S2049: Decode the position vector of the globally optimal individual to obtain the optimal threshold parameter set for image segmentation.

[0034] In one optional implementation, the optimal threshold parameter set is used to drive a pre-built image segmentation model to decompose the image data to be segmented into brushstroke layer data, ink layer data, and stamp layer data, including: S2051: The pre-built image segmentation model is driven by the optimal threshold parameter set. The image segmentation model is built based on an adaptive threshold segmentation algorithm with multi-feature fusion, including a feature preprocessing layer, a parameter mapping layer, a threshold segmentation execution layer, and a post-processing optimization layer. S2052: Use a feature preprocessing layer to preprocess the image data to be segmented to obtain the corresponding gradient feature map, grayscale feature map and hue saturation feature map. Gradient feature map: The "bone method of brushwork" in ink painting is manifested in the image as a sharp change in gray level. The Sobel operator (calculating the derivatives in the X and Y directions and then taking the modulus) can perfectly extract the skeleton of brushstrokes such as flying white and abrupt changes, while forming a natural immunity to large areas of ink color bleeding (low frequency information). Grayscale feature map: Filter out color interference and simply reflect the five color levels of "burnt, dark, heavy, light, and clear"; HSV Hue Saturation Feature Map: In the RGB space, the R channel value of red is also very high, which can easily be confused with ink. After conversion to the HSV space, the vermilion red of the seal is H∈ or [350,360] and has a high saturation S, while the S of ink is extremely low. The two are completely "decoupled" in the feature space. The feature preprocessing layer includes a gradient calculation module, a grayscale conversion module, and an HSV color space conversion module; The gradient calculation submodule calculates the image edge intensity based on the Sobel or Canny operator, and is used to perform gradient calculation on the image data to be segmented, generate gradient feature maps, and capture high-frequency brushstroke information of the painting. The grayscale conversion module is used to convert the image data to be segmented in RGB format into a single-channel grayscale image, generate a grayscale feature map, and analyze the tonal variations of the ink painting. The HSV color space conversion module is used to convert the image data to be segmented in RGB format to HSV space, extract the hue component and saturation component, generate a hue and saturation feature map, and identify specific color (such as red stamp) regions. S2053: Input the optimal threshold parameter set into the parameter mapping layer, and map the optimal threshold parameter set into a specific segmentation decision threshold set that the threshold segmentation execution layer can recognize; The specific segmentation decision threshold set includes a brushstroke segmentation decision threshold, a multi-level ink color segmentation boundary, and a stamp segmentation lower limit threshold. The parameter mapping layer maps the optimal brushstroke segmentation threshold to a gradient binarized brushstroke segmentation decision threshold, maps the ink color segmentation threshold to a multi-level ink color segmentation boundary of a grayscale histogram, and maps the stamp segmentation threshold to a saturation-filtered stamp segmentation lower limit threshold, and locks the target by combining it with a preset red hue range. S2054: Based on the specific segmentation decision threshold set, the threshold segmentation execution layer is used to perform threshold segmentation on the gradient feature map, grayscale feature map and hue saturation feature map to obtain the initial brush stroke bone layer, the initial ink layer and the initial stamp layer. The threshold segmentation execution layer uses a hard threshold binarization algorithm to select pixels in the gradient feature map whose gradient magnitude is greater than the stroke segmentation decision threshold, generating the initial stroke bone layer. A multi-level threshold quantization algorithm is used to divide the grayscale feature map into regions according to the multi-level ink color segmentation boundary, obtaining regions that distinguish the background white space and ink color regions of different shades, generating the initial ink color layer. Finally, a color range masking algorithm is used to extract pixel regions in the hue saturation feature map that simultaneously satisfy the red hue and saturation greater than or equal to the stamp segmentation lower limit threshold in the HSV feature space, obtaining the initial stamp layer. S2055: Using a post-processing optimization layer, connected component analysis and hole filling are performed on the initial brush stroke layer, initial ink layer and initial stamp layer to remove isolated interference pixels and obtain independent brush stroke layer data, ink layer data and stamp layer data. In this embodiment, due to uneven lighting or paper texture, the initial segmentation will inevitably have flaws. For the brushstroke layer, dilation-erosion (opening operation) is used to break up the sticky noise and connect the broken tiny brushstrokes. For the ink layer, the focus is on using "hole filling" (a region growth algorithm based on seed point connected domains) to fill the white holes inside the ink caused by the reflection of Xuan paper fibers. For the stamp layer, isolated red noise with an area smaller than a set pixel value (such as 50 pixels) is removed to obtain clean and independent three-layer data that can be used for rendering.

[0035] In one optional implementation, brushstroke layer data, ink layer data, and stamp layer data are fused and rendered with preset painting analysis information, and then displayed in layers and overlays on an ultra-high-definition large-format display terminal, including: S2061: Pre-build a painting analysis information database to store metadata associated with the large-scale paintings on the ultra-high-definition large-scale display terminal. The metadata includes brushstroke style text descriptions, ink color gradation spectrum and seal interpretation information. S2062: Using the graphics rendering engine of an ultra-high-definition large-format display terminal, multi-layer color mapping and texture generation are performed on the brush stroke layer data, ink layer data and stamp layer data to generate corresponding brush stroke view, ink color view and stamp view. Brush stroke layer rendering: Apply a stroke rendering algorithm to the brush stroke layer data to map areas with a pixel value of 1 to a high-contrast line color (such as dark brown) to generate a brush stroke visual. Ink layer rendering: Based on the levels divided by the ink color segmentation threshold, different areas of the ink color layer data are filled with the corresponding ink color spectrum in the database to restore the ink wash effect and generate an ink color visual image. Seal layer rendering: Fill the seal layer data with vermilion red (RGB: 200, 30, 20) and overlay a semi-transparent shadow effect to enhance the three-dimensionality, generating a seal visual image; S2063: Based on the focus command, the pen stroke view, ink color view and stamp view are displayed in turn in layers until the superposition cycle is reached, and then proceed to the next step; S2064: Using the connected component analysis algorithm, calculate the geometric center coordinates of the brushstroke view, ink color view, and seal view, and call the corresponding painting analysis information in the painting analysis information database to generate a floating UI control with the anchor point coordinates as the geometric center coordinates. S2065: Based on the focus command, a weighted fusion rendering is performed on the pen stroke view, ink color view, and seal view to obtain a rendered view. This rendered view, with floating UI controls, is then displayed on an ultra-high-definition large-format display terminal. The formula is: In the formula, To render the view; The image data to be segmented; These include brushstroke visualizations, ink color visualizations, and seal visualizations. The overall transparency coefficient is superimposed. For fusion weighting coefficients; In this embodiment, the fusion weight coefficient can be dynamically adjusted in actual operation. For example, if a viewer wants to focus on the ink color, they can use gestures to highlight the ink layer. Increase the value to 0.6, and change the original layer 1- The resolution was reduced to 0.2, enabling "interactive on-demand enhanced display," completely changing the pain point of traditional paintings being "visible but not clear."

[0036] This invention also provides a large-scale painting interactive device 300 based on infrared motion capture, see reference. Figure 3 The device may include the following units: The infrared motion capture unit 301 is used to collect user skeletal joint data, calculate the intersection of the hand pointing ray and the screen, and generate a screen interactive cursor coordinate sequence based on an ultra-high-definition large-format display terminal and an infrared motion capture system. The interaction intent recognition unit 302 is used to recognize the motion characteristics of the gesture based on the screen interaction cursor coordinate sequence, determine the interaction intent, and generate corresponding operation instructions. The interactive area locking unit 303 is used to perform stepless zooming and panning of the screen according to the operation command. When the focus command is detected, the image of the current cursor locked area is captured as the image data to be segmented. The segmentation threshold optimization unit 304 is used to call the improved cuckoo catfish optimization algorithm, which initializes the population through chaotic sequences, introduces convergence factors, and integrates the PSO mechanism to perform iterative optimization and output the optimal threshold parameter set for image segmentation. Image segmentation unit 305 is used to drive a pre-built image segmentation model using the optimal threshold parameter set to decompose the image data to be segmented into brush stroke layer data, ink layer data and seal layer data. The fusion rendering unit 306 is used to fuse and render the brush stroke layer data, ink layer data, and seal layer data with the preset painting analysis information, and display them in layers and overlays on the ultra-high-definition giant display terminal.

[0037] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; When the processor executes the program stored in the memory, it implements the infrared motion capture-based interactive method for large-scale paintings of the present invention.

[0038] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EI) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0039] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0040] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the infrared motion capture-based interactive method for large-scale paintings according to embodiments of the present invention.

[0041] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0042] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0043] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0044] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0045] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0046] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for interactive use of large-scale paintings based on infrared motion capture, characterized in that, The method includes: Based on an ultra-high-definition large-format display terminal and an infrared motion capture system, the system collects user skeletal joint data, calculates the intersection of the hand pointing ray and the screen, and generates a sequence of screen interactive cursor coordinates. Based on the screen interaction cursor coordinate sequence, the motion characteristics of the gesture are identified, the interaction intent is determined, and the corresponding operation command is generated. The screen is continuously zoomed and panned according to the operation instructions. When the focus instruction is detected, the image of the currently locked area of ​​the cursor is captured as the image data to be segmented. The improved cuckoo-catfish optimization algorithm is invoked, and the population is initialized by chaotic sequence, a convergence factor is introduced, and the PSO mechanism is fused for iterative optimization, outputting the optimal threshold parameter set for image segmentation. The optimal threshold parameter set is used to drive a pre-built image segmentation model to decompose the image data to be segmented into brush stroke layer data, ink layer data and seal layer data. The brushstroke layer data, ink layer data, and seal layer data are fused and rendered with preset painting analysis information, and then displayed in layers and overlays on an ultra-high-definition giant display terminal.

2. The interactive method for large-scale paintings based on infrared motion capture according to claim 1, characterized in that, The user skeletal joint data includes a sequence of coordinates of key skeletal points in the upper limb and a sequence of coordinates of skeletal joints in the hand. The coordinate sequence of key skeletal points of the upper limb includes the world coordinates of the shoulder joint, elbow joint, and wrist joint at each moment. The hand bone joint coordinate sequence includes the wrist joint coordinates and the set of finger joint coordinates for each finger at each moment.

3. The interactive method for large-scale paintings based on infrared motion capture according to claim 2, characterized in that, Based on an ultra-high-definition large-format display terminal and an infrared motion capture system, the system collects user skeletal joint point data, calculates the intersection of the hand pointing ray and the screen, and generates a sequence of screen-interactive cursor coordinates, including: An infrared motion capture system is used to collect data on the user's skeletal joint points. Using the line connecting the elbow to the wrist as the direction of the interactive ray, and based on the world coordinates of the elbow joint and the world coordinates of the wrist joint in the coordinate sequence of the key bones of the upper limb, a hand pointing ray from the elbow to the wrist is constructed. The intersection of the hand pointing ray and the plane equation of the ultra-high-definition giant display terminal is found. Using the camera calibration matrix, the three-dimensional intersection point is projected onto the two-dimensional coordinate system of the screen to obtain the screen cursor coordinates. Set a timestamp for each screen cursor coordinate to obtain a sequence of screen interactive cursor coordinates that change over time.

4. The interactive method for large-scale paintings based on infrared motion capture according to claim 3, characterized in that, Based on the screen interaction cursor coordinate sequence, the motion characteristics of the gesture are identified, the interaction intent is determined, and corresponding operation instructions are generated, including: Based on the screen interaction cursor coordinate sequence, the motion features of the gesture are extracted, including the cursor displacement vector and the change in hand depth; Based on the hand bone joint coordinate sequence and the motion features, the interaction intent is determined and the corresponding operation command is generated. The interaction intent includes translation intent, zoom intent and focus intent, and the corresponding operation command includes translation command, zoom command and focus command.

5. The interactive method for large-scale paintings based on infrared motion capture according to claim 4, characterized in that, Based on the hand bone joint coordinate sequence and the motion characteristics, the interaction intent is determined and corresponding operation instructions are generated, including: The hand bone joint coordinate sequence is input into a pre-trained gesture recognition model to obtain the corresponding gesture recognition results. The gesture recognition results include fist gestures, open gestures, and push gestures. The gesture recognition model is constructed based on the LSTM algorithm. If the gesture recognition result is a clenched fist gesture, and the absolute value of the cursor displacement vector in the motion features is greater than the displacement threshold, then the interaction intent is determined to be a translation intent, and the corresponding operation instruction is a translation instruction. If the gesture recognition result is an open gesture, and the rate of change of the distance between the hands is greater than the rate of change threshold, then the interaction intent is determined to be a scaling intent, and the corresponding operation instruction is a scaling instruction. If the gesture recognition result is a pushing gesture, and the change in hand depth in the motion feature is greater than the change threshold within a preset continuous time window, and the cursor displacement vector of the screen cursor coordinates within a preset displacement tolerance radius centered on the initial screen cursor coordinates within the time window does not exceed the area, then the interaction intent is determined to be a focus intent, and the corresponding operation instruction is generated as a focus instruction.

6. The interactive method for large-scale paintings based on infrared motion capture according to claim 5, characterized in that, The screen is continuously zoomed and panned according to the operation instructions. When a focus instruction is detected, the image of the currently locked cursor area is captured as the image data to be segmented, including: If the operation command is a panning command or a zoom command, then the ultra-high-definition large-format display terminal will perform stepless zooming and panning of the screen according to the operation command. If the operation command is a focus command, then the specific area of ​​the giant painting on the ultra-high-definition giant display terminal with the screen cursor coordinates is taken as the current cursor lock area, and the screen cursor coordinates are taken as the center coordinates. Based on the center coordinates, the positioning rectangle area of ​​the ultra-high-definition giant display terminal is determined, and based on the positioning rectangle area, the image of the currently cursor-locked area is extracted as the image data to be segmented.

7. The interactive method for large-scale paintings based on infrared motion capture according to claim 6, characterized in that, An improved cuckoo-catfish optimization algorithm is applied, which initializes the population with a chaotic sequence, introduces a convergence factor, and incorporates a PSO mechanism for iterative optimization, outputting the optimal threshold parameter set for image segmentation, including... The segmentation threshold parameter vector of the image data to be segmented is encoded as the position vector of an individual in the improved cuckoo catfish optimization algorithm. A fitness function is set, and historical image data is used as a simulation environment. The segmentation threshold parameter vector includes brush stroke segmentation threshold, ink color segmentation threshold, and seal segmentation threshold. The chaotic sequence is generated using the Logistic mapping, and then mapped to the solution space of individuals in the improved cuckoo-catfish optimization algorithm to obtain the initial population. Using the fitness function, calculate the fitness value of each initial individual in the initial population, and determine the optimal position of each initial individual and the global optimal individual in the initial population based on the fitness value. In the early stages of iteration, a Levy flight mechanism is introduced to perform position updates of cuckoo nest parasitism on the initial population or the population updated in the last iteration, resulting in an updated population. In the middle of the iteration, based on the individual optimal position and the global optimal individual, the PSO mechanism and random perturbation are introduced to perform position updates of the catfish defense behavior on the population updated in the last update, and the updated population is obtained. In the later stages of the iteration, based on the individual optimal position and the global optimal individual, the population updated in the previous update is subjected to a fine-grained search behavior dominated by the PSO mechanism to obtain the updated population. Using the fitness function, calculate the fitness value of each updated individual in the updated population, and update the global best individual based on the fitness value; Repeatedly update the position of the population. When the number of iterations reaches the maximum number of iterations or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the globally optimal individual. The position vector of the globally optimal individual is decoded to obtain the optimal threshold parameter set for image segmentation.

8. The interactive method for large-scale paintings based on infrared motion capture according to claim 7, characterized in that, Using the optimal threshold parameter set to drive a pre-built image segmentation model, the image data to be segmented is decomposed into brushstroke layer data, ink layer data, and stamp layer data, including: The optimal threshold parameter set is used to drive a pre-built image segmentation model, which is built based on an adaptive threshold segmentation algorithm with multi-feature fusion, including a feature preprocessing layer, a parameter mapping layer, a threshold segmentation execution layer, and a post-processing optimization layer. The feature preprocessing layer is used to preprocess the image data to be segmented to obtain the corresponding gradient feature map, grayscale feature map and hue saturation feature map. The optimal threshold parameter set is input to the parameter mapping layer, and the optimal threshold parameter set is mapped to a specific segmentation decision threshold set that the threshold segmentation execution layer can recognize; Based on the specific segmentation decision threshold set, a threshold segmentation execution layer is used to perform threshold segmentation on gradient feature map, grayscale feature map and hue saturation feature map to obtain the initial brush stroke bone layer, initial ink layer and initial seal layer. Using a post-processing optimization layer, connected component analysis and hole filling are performed on the initial brush stroke layer, initial ink layer, and initial stamp layer to remove isolated interference pixels and obtain independent brush stroke layer data, ink layer data, and stamp layer data.

9. The interactive method for large-scale paintings based on infrared motion capture according to claim 8, characterized in that, The brushstroke layer data, ink layer data, and seal layer data are fused and rendered with preset painting analysis information, and then displayed in layers and overlays on an ultra-high-definition large-format display terminal, including: A database of artwork analysis information is pre-built to store metadata associated with the large-scale artworks on the ultra-high-definition large-scale display terminal. The metadata includes text descriptions of brushstroke styles, color gradations of ink, and seal interpretation information. Using the graphics rendering engine of the ultra-high-definition large-format display terminal, multi-layer color mapping and texture generation are performed on the brush stroke layer data, ink layer data and seal layer data to generate corresponding brush stroke view, ink color view and seal view. Based on the focus command, the pen stroke view, ink color view and seal view are displayed in turn in layers until the superposition cycle is reached, and then the next step is entered. Using a connected component analysis algorithm, the geometric center coordinates of the brushstroke view, ink color view, and seal view are calculated, and the corresponding painting analysis information is called from the painting analysis information database to generate a floating UI control with the anchor point coordinates as the geometric center coordinates. Based on the focus command, the pen stroke view, ink color view, and seal view are weighted and fused to obtain a rendered view. The rendered view with floating UI controls is then displayed on an ultra-high-definition large-format display terminal.

10. A large-scale painting interaction device based on infrared motion capture, used to implement the large-scale painting interaction method based on infrared motion capture as described in any one of claims 1-9, characterized in that, The device includes: The infrared motion capture unit is used to collect user skeletal joint data, calculate the intersection of the hand pointing ray and the screen, and generate a screen interactive cursor coordinate sequence based on an ultra-high-definition large-format display terminal and an infrared motion capture system. An interaction intent recognition unit is used to identify the motion characteristics of gestures based on the screen interaction cursor coordinate sequence, determine the interaction intent, and generate corresponding operation instructions. The interactive region locking unit is used to perform stepless zooming and panning of the screen according to the operation command. When the focus command is detected, the image of the current cursor locked area is captured as the image data to be segmented. The segmentation threshold optimization unit is used to call the improved cuckoo-catfish optimization algorithm, which initializes the population through chaotic sequences, introduces convergence factors, and integrates the PSO mechanism to perform iterative optimization and outputs the optimal threshold parameter set for image segmentation. The image segmentation unit is used to drive a pre-built image segmentation model using the optimal threshold parameter set to decompose the image data to be segmented into brush stroke layer data, ink layer data and seal layer data. The fusion rendering unit is used to fuse and render brush stroke layer data, ink layer data, and seal layer data with preset painting analysis information, and then display them in layers and overlays on an ultra-high-definition large-format display terminal.

Images