Wireless induction-based interactive projection control method and system, and storage medium

Abstract

The application relates to a wireless induction-based interactive projection control method and system and a storage medium, and relates to the technical field of interactive projection, which comprises the following steps: collecting a shooting position of a camera in a venue and detection image information shot by the shooting position; obtaining a visual blind area based on the detection image information and a preset venue specification; identifying a mobile path of an interactive user and the interactive user from the detection image information; generating a scheduling signal of a drone when the mobile path is directed to the visual blind area; obtaining a blind area volume based on the visual blind area and the venue specification in response to the scheduling signal; obtaining a refraction position based on the blind area volume and the shooting position, and controlling the drone to fly to the refraction position to shoot a supplementary visual field image of the visual blind area; obtaining interactive images through the supplementary visual field image, the detection image information and the interactive user, and outputting the interactive images. The application has the effect of improving the accuracy of interactive image output.

Description

Technical Field

[0001] This invention relates to the technical field of interactive projection, and in particular to an interactive projection control method, system, and storage medium based on wireless sensing. Background Technology

[0002] Interactive projection is an interactive technology that captures users' body movements and changes the projected image in real time. It is widely used in public places such as science and technology museums, exhibition halls, shopping malls, and amusement parks.

[0003] In the process of interactive projection, images are usually captured in the venue where the interactive projection is required. Fixed cameras installed in the venue capture images of users in the projection area, and the user images are analyzed to see the actions that users need to interact with. The actions are combined with the objects that users need to interact with in the venue to generate a projection, and the resulting projection is output using a projector.

[0004] During interactive projection, the field of view of a fixed camera is limited by factors such as its installation location, the distribution of obstacles in the venue, and interference from projection light, resulting in blind spots. Even after adding cameras, blind spots still exist due to mutual obstruction between people, and the camera cannot fully capture the user's movements, leading to deviations in the interactive images. Summary of the Invention

[0005] To improve the accuracy of interactive image output, this invention provides an interactive projection control method, system, and storage medium based on wireless sensing.

[0006] In a first aspect, the present invention provides an interactive projection control method based on wireless sensing, which adopts the following technical solution: An interactive projection control method based on wireless sensing, comprising: Collect the shooting positions of cameras within the venue and the detection image information captured by those shooting positions; The blind spot is determined based on the detected image information and the preset venue specifications; Identify the interactive users and their movement paths from the detected image information; When the movement path is directed toward the blind spot, a dispatch signal for the drone is generated; In response to scheduling signals, the volume of the blind spot is obtained based on the blind spot and the venue specifications; The refraction position is obtained based on the volume of the blind spot and the shooting position, and the drone is controlled to fly to the refraction position to take pictures to obtain supplementary field-of-view images of the blind spot; Interactive images are obtained by supplementing the field of view image, detecting image information, and interacting with the user, and then output as interactive images.

[0007] By adopting the above technical solution, the system analyzes the shooting positions, detected image information, and movement paths of cameras within the venue to predict the trend of users entering blind spots and generates drone scheduling signals to control drones to fly to the refraction position to obtain supplementary field-of-view images. This dynamically expands the field of view of fixed cameras and proactively deploys supplementary observation nodes before users enter blind spots. Even if the original fixed cameras have blind spots caused by installation positions, obstacles, or crowds, the system can still obtain accurate and complete user action information, ensuring the continuity and real-time response of interactive projection, reducing interactive interruptions or image deviations, and improving the reliability and accuracy of interactive projection.

[0008] Optionally, methods for obtaining interactive images include: The action and detection information of the person's image are obtained by detecting image information and supplementing visual field images; The detection acceleration and detection inertial force are used to collect detection information; By combining detected acceleration, detected inertial force, and human image motion, the detected and predicted motion is obtained; The orientation of the field of view is determined based on the detected image information and the detection information. The objects observed are determined based on the orientation of the field of view and the specifications of the venue. Interactive images are created by observing objects and detecting and predicting actions.

[0009] By adopting the above technical solution, the human image action is obtained by fusing detection image information and supplementary visual field image, and the user's action is predicted by combining detection acceleration and detection inertial force. At the same time, the observed object is determined based on the visual field orientation. Thus, the detected and predicted action and the observed object are combined to form an interactive image, which can predict the user's interaction intention in advance even in the blind spot of the visual field, reduce the action recognition delay caused by image loss, and improve the response speed and accuracy of interactive images.

[0010] Optionally, methods for obtaining interactive images include: Based on the test information, cooperation information is obtained; When interactive users exhibit cooperative information, the number of users whose cooperative information is detected is identified from the supplementary visual field image. Retrieve the baseline number of participants from the coordination information; The number of people whose images were blocked was determined by comparing the baseline number of people with the number of people tested. Occlusion information is obtained based on supplementary visual images, the number of people whose vision is obstructed, and cooperation information; The detection acceleration and detection inertial force of occlusion information are used as the occlusion acceleration and occlusion inertial force; Occlusion-based interactive projection is obtained by considering occlusion acceleration, occlusion inertial force, and refraction position. Interactive projection is obtained by combining the detected and predicted actions with the occlusion interactive projection.

[0011] By adopting the above technical solution, the detection information is used to extract the cooperation information and identify the number of people being detected. Then, the number of people being occluded and the occlusion information are determined by comparing the baseline number of people with the number of people being detected. Then, the acceleration and inertial force in the occlusion information are used to generate the occlusion interactive projection, and the interactive projection is obtained by fusing it with the detection and prediction actions. In this way, even when multiple people are occluding each other, the action information of the occluded users can still be completed, realizing complete interactive projection in multi-person scenarios and reducing the situation of interaction omissions or errors caused by human occlusion.

[0012] Optionally, methods for obtaining occlusion-based interactive projections include: The detection and identification position corresponding to the detected acceleration based on the collected occlusion information; A detection and identification trajectory is formed based on the detection and identification location; Occlusion interaction actions are obtained by detecting and recognizing the trajectory, occlusion acceleration, and occlusion inertial force; The occlusion interaction projection is obtained based on the occlusion interaction action and the refraction position.

[0013] By adopting the above technical solution, the occlusion interaction action is obtained by analyzing the detection and recognition position, occlusion acceleration and occlusion inertial force. The occlusion interaction action and refraction position are combined to generate the occlusion interaction projection. Thus, without relying on the complete image, the real-time action of the occluded user can be reconstructed by using the occlusion inertial force and the detection and recognition trajectory, thereby improving the robustness of action recognition and the accuracy of the projected content in occluded scenes.

[0014] Optionally, methods for obtaining occlusion-based interactive projections include: The occlusion range is obtained based on supplementary visual images and coordination information; The occlusion execution range is obtained by combining occlusion information with occlusion interaction actions. The offset distance is obtained by combining the occlusion execution range with the corresponding occlusion range; The offset angle is obtained based on the offset distance and refraction position, and the mirror preset on the drone is controlled to rotate at the offset angle, and the supplementary field of view image is updated; Occlusion interaction projection is obtained based on the updated supplementary field of view image and occlusion interaction actions.

[0015] By adopting the above technical solution, the occlusion range is determined by supplementing the field of view image and the cooperation information. Then, the occlusion execution range is obtained by combining the occlusion interaction action, and the offset distance is calculated to obtain the mirror offset angle. The mirror on the UAV is controlled to rotate to update the supplemented field of view image, thereby obtaining a more accurate occlusion interaction projection. By dynamically adjusting the reflective mirror, the occlusion area is actively avoided, the observation angle and projection optical path of the UAV are optimized, and the capture accuracy of the occluded user's actions and the integrity of the interactive projection are improved.

[0016] Optionally, methods for obtaining interactive projections include: Update the occlusion interaction action to update the interaction projection, and define the updated interaction projection as the calibration interaction projection; The baseline variation parameters are obtained by comparing the interactive projection with the calibrated interactive projection. Retrieve the original illumination parameters based on the interactive projection; The refraction range is obtained by comparing the refraction position with the preset mirror specifications; The original refraction parameters for each refraction range are obtained based on the reference variation parameters and the original irradiation parameters; The calibration tilt angle is obtained by combining the original illumination parameters, the original refraction parameters, the reference change parameters, and the occlusion interaction actions, and the mirror on the UAV is controlled to adjust the calibration tilt angle.

[0017] By adopting the above technical solution, the calibrated interactive projection is obtained by updating the occlusion interaction action, the reference change parameters are obtained by comparing the before and after projection, and the original refraction parameters in each refraction range are obtained based on the original illumination parameters and refraction range. Then, the calibrated tilt angle is solved by combining the occlusion interaction action to adjust the mirror, thereby realizing the closed-loop optimization of the mirror angle. It can reduce the amount of change in the projector adjustment by using mirror reflection while meeting the user's action recognition requirements.

[0018] Optionally, methods for obtaining the calibration tilt angle include: The constant original parameters are obtained by removing the refraction parameters from the original irradiation parameters; The remaining interactive projection is obtained by removing the constant original parameters from the calibration interactive projection and then performing projection changes based on the corresponding reference change parameters. The tilt angle is determined based on the remaining interactive projection, the original refraction parameters, and the occlusion interactive action.

[0019] By adopting the above technical solution, the refraction parameters are removed from the original illumination parameters to obtain constant original parameters. Then, the constant original parameters are removed from the calibration interactive projection to obtain the remaining interactive projection. Based on the remaining interactive projection, the refraction parameters, and the occlusion interactive action, the calibration tilt angle is solved. This decouples the projection content that must be completed by mirror refraction from the fixed projection content, simplifies the optimization calculation of the mirror angle, and improves the system adjustment efficiency and projection quality.

[0020] Optionally, methods for obtaining the calibrated tilt angle also include: Action recognition feature points are obtained based on occlusion-interaction actions; The range of action execution is obtained by recognizing feature points and detecting the recognition trajectory. The offset distance is obtained by combining the action execution range with the occlusion range. The refraction range of the marker is obtained by defining each refraction range based on the offset distance of the marker; Based on the overlap between the residual interactive projection and the original refraction parameters, the non-overlapping original calibration parameters and residual calibration projection are obtained. The deviation variation parameters are obtained by calibrating the original parameters, calibrating the remaining projection, and marking the refraction range. The marked refraction range corresponding to the minimum deviation change parameter is taken as the target refraction range, and the calibrated tilt angle is obtained based on the target refraction range.

[0021] By adopting the above technical solution, the action recognition feature points are extracted by occlusion interaction actions and the action execution range is obtained by combining the detection and recognition trajectory. The mark offset distance is obtained by comparing with the corresponding occlusion range to limit the mark refraction range. Then, the calibration tilt angle is obtained by the coincidence analysis of the remaining interactive projection and the original refraction parameters. Thus, the optimal mirror angle is selected from multiple candidate mirror angles to minimize the projection deviation, thereby realizing the fine and intelligent adjustment of the mirror angle and improving the adaptability of interactive projection in complex occlusion scenarios.

[0022] Secondly, this application provides an interactive projection control system based on wireless sensing, which adopts the following technical solution: A wireless sensing-based interactive projection control system includes: The acquisition module is used to acquire the shooting location and detection image information; A memory for storing the program of the wireless sensing-based interactive projection control method; The processor is used to load and execute programs stored in memory.

[0023] Thirdly, this application provides a computer storage medium capable of storing corresponding programs, which facilitates improving the accuracy of interactive projection, and adopts the following technical solution: A computer-readable storage medium storing a computer program capable of being loaded by a processor and executed for a wireless sensing-based interactive projection control method.

[0024] In summary, this application includes at least one of the following beneficial technical effects: 1. By analyzing the shooting positions, detected image information, and movement paths of cameras within the venue, the system predicts the trend of users entering blind spots and generates drone scheduling signals to control drones to fly to the refraction position to obtain supplementary field-of-view images. This dynamically expands the field of view of fixed cameras and proactively deploys supplementary observation nodes before users enter blind spots. Even if the original fixed cameras have blind spots caused by installation positions, obstacles, or crowds, the system can still obtain accurate and complete user action information, ensuring the continuity and real-time response of interactive projection, reducing interactive interruptions or image deviations, and improving the reliability and accuracy of interactive projection. 2. By supplementing the field of view image and cooperating information, the occlusion range is determined. Then, the occlusion execution range is obtained by combining the occlusion interaction action. The offset distance is calculated to obtain the mirror offset angle. The mirror on the UAV is controlled to rotate to update the supplemented field of view image, thereby obtaining a more accurate occlusion interaction projection. By dynamically adjusting the reflective mirror, the occlusion area is actively avoided, the observation angle and projection optical path of the UAV are optimized, and the capture accuracy of the occluded user's actions and the integrity of the interactive projection are improved. 3. By extracting action recognition feature points through occlusion interaction actions and combining them with the detected recognition trajectory to obtain the action execution range, and comparing it with the corresponding occlusion range to obtain the marker offset distance to limit the marker refraction range, the calibration tilt angle is obtained by analyzing the overlap between the remaining interactive projection and the original refraction parameters. Thus, the optimal mirror angle is selected from multiple candidate mirror angles to minimize the projection deviation, thereby achieving fine and intelligent adjustment of the mirror angle and improving the adaptability of interactive projection in complex occlusion scenarios. Attached Figure Description

[0025] Figure 1 This is a flowchart of an interactive projection control method based on wireless sensing according to an embodiment of the present invention; Figure 2 This is a flowchart of a method for obtaining interactive images according to an embodiment of the present invention; Figure 3 This is a flowchart of the method for obtaining interactive images according to an embodiment of the present invention. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0027] Reference Figure 1 This application discloses an interactive projection control method based on wireless sensing, comprising the following steps: S10: Collect the shooting positions of the cameras inside the venue and the detection image information captured by the shooting positions.

[0028] The shooting position refers to the installation coordinates and orientation angle of the fixed camera in the venue in three-dimensional space. The shooting position is obtained by retrieving the installation coordinates of the fixed camera in the venue from the system.

[0029] The detection image information refers to the images of the area to be interactively projected, captured by fixed cameras within the venue. These images are used as the detection image information.

[0030] S11: Obtain the blind spot based on the detected image information and the preset venue specifications.

[0031] The venue specifications are the parameters set by the technicians, such as the dimensions of the venue, the boundaries of the projection area, and the distribution of obstacles.

[0032] The blind spot refers to the spatial area in the detected image that has not been captured. By combining the location of obstacles in the venue specifications with the field of view of the fixed camera, the spatial area that the image has not captured is obtained through geometric calculation as the blind spot.

[0033] S12: Identify the interactive user and their movement path from the detected image information.

[0034] Interactive users refer to users who need to interact with the projection in the detected image information. The human body detection algorithm identifies the people in the detected image information and matches them with the people recorded in the system to obtain their corresponding identity information as interactive users. In this embodiment, before visiting the venue, it is necessary to record the basic information of the person, such as identity identification, height and weight, and obtain the user's authorization.

[0035] The movement path refers to the path that an interactive user moves through the detected image information. It is formed by identifying the position of the interactive user from the detected image information, recording each position in multiple consecutive frames of images, and connecting them to form the movement path.

[0036] S13: When the movement path is towards the blind spot, generate a scheduling signal for the drone.

[0037] The dispatch signal is a signal set by technicians to call upon the drone for assistance. When the movement path is towards the blind spot, it means that the user cannot recognize the projection interaction when needed in the blind spot. In this case, the dispatch signal is output to the system to call upon the drone to fly.

[0038] S14: In response to the scheduling signal, the blind spot volume is obtained based on the blind spot and the venue specifications.

[0039] The blind spot volume refers to the spatial volume occupied by the blind spot. It is calculated by projecting the blind spot onto a horizontal ground and integrating the volume values ​​at different vertical heights within the blind spot.

[0040] S15: Based on the volume of the blind spot and the shooting position, the refraction position is obtained, and the drone is controlled to fly to the refraction position to shoot and obtain a supplementary field-of-view image of the blind spot.

[0041] The refraction position refers to the position where the mirror carried under the drone needs to be used for refraction. By taking the center point of the blind zone volume as the reflection target point, and combining the shooting position of the fixed camera with the law of light reflection, a system of equations is established based on the fixed camera position, the center point of the blind zone volume, and the normal constraint of the mirror to solve for the coordinates of the hovering point in the air to obtain the refraction position. This refraction position enables the camera at the shooting position to obtain the content in the blind zone of the field of view refracted in the mirror when shooting at the mirror.

[0042] Supplemental field-of-view images refer to images of the blind spots in the field of view obtained by photographing a mirror. By controlling a drone to fly to the refraction position and using the camera at the corresponding shooting position calculated by the refraction position to photograph the mirror, the image obtained from photographing the mirror is used as a supplemental field-of-view image.

[0043] Establish and solve the system of equations: Let the camera position be A, the center point of the blind zone volume be B, the drone hovering point (center of the mirror) be C, and the mirror normal be N. The vector relationship that satisfies the angle of incidence equals the angle of reflection is: (AC) / |AC| and (BC) / |BC| are symmetric about the normal N. When the initial orientation of the mirror is preset to be parallel to the ground or perpendicular to the camera's optical axis, the unique or optimal solution for point C can be obtained by solving the above constraint equations.

[0044] In this embodiment, the drone can also be equipped with two independent and controllable reflective mirrors: the first mirror is used to assist the fixed camera in capturing blind spots, and the second mirror is used to assist the projector in reflecting interactive images to the obscured user. Alternatively, the same mirror can be used in a time-division multiplexing manner, serving both shooting and projection tasks at different times, or performing partial projection while continuously capturing user movements.

[0045] S16: Interactive images are obtained by supplementing the field of view image, detecting image information, and interacting with the user, and then output as interactive images.

[0046] Interactive images refer to two-dimensional or three-dimensional images that interact with users. They are obtained by analyzing supplementary field-of-view images, detected image information, and user interactions, and are then used to control a corresponding image generation device for output. Specific analysis methods for interactive images are described below.

[0047] Reference Figure 2 Methods for obtaining interactive images include: S20: Obtain the character image's motion and detection information by detecting image information and supplementing the field of vision image.

[0048] Human image motion refers to the user's limb movement posture extracted from the detection image information and the supplementary field of view image. By detecting the human skeleton key points in the detection image information and the supplementary field of view image respectively, and based on the relative pose of the fixed camera and the drone, the coordinate system of the supplementary field of view image is transformed into the coordinate system of the detection image to form a continuous sequence of motion as human image motion.

[0049] The detection information refers to the identity identifier and associated sensor device information (such as wristband ID, accelerometer model, etc.) of the interactive user pre-registered in the system. The interactive user is supplemented by the detection image information and the supplementary field of view image. After the supplementation is completed, the corresponding identity identifier and sensor device information of all interactive users are retrieved from the system as the detection information.

[0050] S21: Detection acceleration and detection inertial force for collecting detection information.

[0051] Detected acceleration refers to the linear acceleration vector (including x, y, and z axis components) measured in real time by the wristband worn by the interactive user. The acceleration data from the acceleration sensor in the wristband corresponding to the detection information is used as the detected acceleration.

[0052] The detected inertial force refers to the inertial force vector calculated from the detected acceleration and the user's hand mass. The velocity change is obtained by integrating the collected detected acceleration over time. Combined with the body posture data in the detected information, the user's hand mass is obtained and the inertial force is calculated as the detected inertial force.

[0053] Detected inertial force = User's hand mass × Detected acceleration.

[0054] S22: Combine detected acceleration, detected inertial force, and human image motion to obtain detected and predicted motion.

[0055] Action detection and prediction refers to predicting the limb movements a user will make in the next moment (e.g., 0.1 seconds in the future). By taking the image of the person's motion, detected acceleration, and detected inertial force as input, and using a Kalman filter or a short time memory neural network (LSTM) model, the output is the coordinates of the user's skeleton key points or the action category at the future moment, which is used as the action detection and prediction.

[0056] In this embodiment, the Kalman filter or Short Time Memory Neural Network (LSTM) model is pre-trained using a large amount of motion data. The motion data of a person's image (represented as a 3D coordinate sequence of 20 joints), the three-axis detection acceleration sequence, and the detection inertial force sequence from the past 30 frames within the last second are time-aligned and concatenated to form a multi-dimensional time series feature matrix. This matrix is ​​then input into a network consisting of two layers of 128 LSTM units. The network outputs the predicted 3D coordinates of the user's 20 joints within the next 0.2 seconds as the predicted motion.

[0057] S23: Obtain the field of view orientation based on the detected image information and detection information.

[0058] Visual field orientation refers to the current gaze direction or facial orientation of the interacting user. Facial features of the interacting user are extracted from the detected image information, and the direction the user is facing is determined as the visual field orientation using pupil direction or head pose estimation algorithms.

[0059] S24: Observed items are obtained based on the orientation of the field of view and the specifications of the venue.

[0060] The observed object refers to an object or virtual interactive element in the projection area that the interactive user is currently looking at. Using the direction of the viewpoint as the ray direction and combining the spatial coordinates of the projected object in the venue specifications, the object that intersects with the line of sight and is closest to the object is identified as the observed object through the intersection detection of the ray and the object.

[0061] S25: Interactive images are formed by observing objects and detecting and predicting actions.

[0062] The system takes the identification of observed objects and the action type of detected and predicted actions as input, selects corresponding 3D animation models from a preset interactive content library, and generates a sequence of projected images synchronized with the user's actions through real-time rendering as interactive images.

[0063] The interactive content library is a pre-defined, stored database of content that can interact with users, pre-configured by technical personnel. The interactive content library stores content in key-value pairs: the key is (observed item ID, predicted action type), and the value is the path to the corresponding 3D animation model file. For example, ("virtual button", "click") corresponds to an animation of a button being pressed. Once the observed item ID and predicted action type are obtained, an exact match is performed to retrieve the content from the interactive content library; if the content does not exist, the default animation is used.

[0064] Reference Figure 3 The methods for obtaining interactive images include: S30: Obtain coordination information based on the test information.

[0065] Coordination information refers to the set of information about the interactive user and their fellow users participating in the interactive projection, including the companions' identities, number of people, spatial location, and relationships (parent-child relationships, etc.). The team identifier of the interactive user (such as the same visitor group number or the type of identity coordination) is extracted from the detection information. The identities and sensor information of all members under that team identifier are retrieved from the system and combined with the detection information to form the coordination information.

[0066] S31: When interactive users have cooperation information, identify the number of people whose cooperation information is detected from the supplementary visual field image.

[0067] The number of people detected refers to the number of people in the supplementary visual field image whose cooperation information is located within the range. When there is cooperation information among interactive users, it indicates that there are many people moving into the blind spot of the visual field and that there is mutual occlusion between people. The range where the cooperation information is located is selected by using a preset reference spacing, and the number of people in the range identified from the supplementary visual field image is used as the number of people detected.

[0068] The baseline spacing is a distance set by technicians to determine the distance between people visiting a venue together.

[0069] S32: Retrieve the baseline number of people from the coordination information.

[0070] The baseline number of users refers to the total number of interactive users and companion users that should theoretically exist in the coordination information. The baseline number of users is obtained by retrieving the coordination information.

[0071] S33: The number of people obscured is obtained by comparing the baseline number of people with the number of people being tested.

[0072] The number of people obscured refers to the deviation between the baseline number and the number of people being tested. By analyzing the consistency between the baseline number and the number of people being tested, it is found that there is no obstruction between people when the baseline number and the number of people being tested are consistent, and no adjustment is made.

[0073] When the baseline number of people is inconsistent with the number of people being tested, it indicates that there is occlusion between people. In this case, the difference between the baseline number of people and the number of people being tested is calculated as the number of people with occlusion.

[0074] S34: Obstruction information is obtained based on supplementary visual field images, the number of people obstructed, and coordination information.

[0075] Occlusion information refers to the information of the companion user that is occluded. By identifying the information of each user within the range of the cooperation information from the supplementary visual field image, and removing the cooperation information, the remaining cooperation information is taken as occlusion information.

[0076] S35: Use the detection acceleration and detection inertial force of occlusion information as the occlusion acceleration and occlusion inertial force.

[0077] Occlusion acceleration refers to the linear acceleration vector measured by the wristband worn by the companion user who is being occluded. The occlusion acceleration is obtained by retrieving the detection acceleration on the wristband corresponding to the occlusion information.

[0078] Occlusion inertial force refers to the inertial force vector generated by the hand of the companion user that is being occluded. It is obtained by integrating the occlusion acceleration over time to obtain the velocity change, and combined with the user's body posture data in the occlusion information, it is input into the user model to obtain the hand mass. The hand mass × occlusion acceleration = inertial force is calculated and used as the occlusion inertial force.

[0079] S36: Occlusion interactive projection is obtained by occlusion acceleration, occlusion inertial force and refraction position.

[0080] Occlusion-based interactive projection refers to the interactive projection screen generated for a fellow user whose view is obstructed. It is derived by analyzing the occlusion acceleration, occlusion inertial force, and refraction position. The specific analysis method for occlusion-based interactive projection is described below.

[0081] S37: Combine the detected and predicted action with the occlusion interactive projection to obtain the interactive projection.

[0082] By aligning the interactive images corresponding to the detected and predicted actions with the occlusion interactive projections in time and space, they are merged into the same projection screen or combined into a continuous multi-person coordinated action projection output to the projector for projection.

[0083] Methods for obtaining occlusion-based interactive projections include: S40: The detection and identification position of the acceleration corresponding to the occlusion information collected.

[0084] The detection and identification position refers to the three-dimensional spatial position of the wristband that is obscuring information. This is achieved by retrieving the positioning system of the corresponding sensor from the obscuring information and using the three-dimensional spatial coordinate data collected by the positioning system as the detection and identification position.

[0085] S41: Form a detection and recognition trajectory based on the detection and recognition location.

[0086] The detection and recognition trajectory refers to the trajectory of a user's hand movement that obscures information. The trajectory formed by connecting multiple consecutive detection and recognition positions is used as the detection and recognition trajectory.

[0087] S42: Obstruction interaction actions are obtained by detecting and recognizing the trajectory, occlusion acceleration, and occlusion inertial force.

[0088] Occlusion-based interactive actions refer to the predicted limb movements that users need to interact with when information is occluded. By using the detected and recognized trajectory as a position constraint and the occlusion acceleration and occlusion inertial force as motion features, a Kalman filter or a short-time memory neural network (LSTM) model is used for fusion estimation. The output is the expected action posture of the occluded user at the next moment (such as the coordinates of the skeleton key points) as the occlusion-based interactive action.

[0089] S43: Obtain the occlusion interaction projection based on the occlusion interaction action and the refraction position.

[0090] The occlusion interaction projection is obtained by analyzing the occlusion interaction action and the refraction position. The specific analysis method for the occlusion interaction projection is described below.

[0091] Methods for obtaining occlusion-based interactive projections include: S50: Obtain the occlusion range based on the supplementary field image and occlusion information.

[0092] The occlusion range refers to the spatial area where companion users in the cooperation information mutually occlude each other in the supplementary visual field image. The contours of visible users in all cooperation information are identified from the supplementary visual field image. The bounding contour of each cooperation information is calculated. The overlapping area of ​​multiple bounding contours on the image plane and the range of their outward extension by a preset threshold (e.g., 5 cm) are taken as the cooperation occlusion range.

[0093] S51: Obtain the occlusion execution range through occlusion information and occlusion interaction actions.

[0094] The occlusion execution range refers to the area in space occupied by a user's body or hand when the user interacts with an occlusion interaction action. The skeletal keypoint coordinates in the occlusion interaction action are mapped to the actual coordinate system. Then, the body posture data from the occlusion information is input into the user model to obtain the limb length. The union of spherical regions centered on each keypoint and with a radius equal to the limb length (e.g., 20 cm) is calculated as the occlusion execution range.

[0095] S52: The offset distance is obtained by combining the occlusion execution range with the matching occlusion range.

[0096] Offset distance refers to the spatial distance required to prevent the occlusion execution range from overlapping with the cooperating occlusion range. It is calculated as the minimum Euclidean distance in the horizontal direction between the occlusion execution range and the cooperating occlusion range. In this embodiment, this offset distance needs to be converted between the camera and the actual location.

[0097] S53: Based on the offset distance and refraction position, the offset angle is obtained, and the mirror preset on the drone is controlled to rotate at the offset angle, and the supplementary field of view image is updated.

[0098] The offset angle refers to the angle at which the reflective mirror on the drone needs to rotate so that the supplementary field-of-view image can capture the person obstructing the view, thereby calibrating the occlusion interaction action. By taking the refraction position as the vertex and the current normal direction of the mirror as the reference, the offset distance (i.e., the spatial offset between the center of the current mirror reflection spot and the expected position of the obstructed user) and the relative distance between the drone and the obstructed user are used to calculate the pitch and yaw angles that the mirror needs to deflect as the offset angle through geometric relationships (such as triangulation or the law of mirror reflection). The mirror on the drone is then controlled to rotate at the offset angle. After the mirror completes the angle rotation, the supplementary field-of-view image is re-captured.

[0099] S54: Obtain the occlusion interaction projection based on the updated supplementary field of view image and the occlusion interaction action.

[0100] The occlusion interaction projection is obtained by analyzing the updated supplementary field of view image and occlusion interaction action with reference to S22 to S25.

[0101] Methods for obtaining interactive projections include: S60: Update the occlusion interaction action to update the interaction projection, and define the updated interaction projection as the calibration interaction projection.

[0102] The calibration interactive projection refers to the updated interactive projection, which obtains the occlusion interaction action again and repeats S42 to S43 to generate a new interactive projection using the updated occlusion interaction action, and defines the updated interactive projection as the calibration interactive projection.

[0103] S61: Compare the interactive projection with the calibration interactive projection to obtain the baseline variation parameters.

[0104] The baseline variation parameter is the overall change between the calibrated interactive projection and the calibrated interactive projection, including the magnitude of changes in image content, pixel differences, or changes in luminous flux. By calculating the inter-frame difference between the interactive projection and the calibrated interactive projection, the grayscale mean, variance, or structural similarity (SSIM) of the difference image is extracted as the baseline variation parameter.

[0105] S62: Retrieve the original illumination parameters based on the interactive projection.

[0106] The original illumination parameters refer to the light source emission direction, emission position, and light intensity distribution of the projector used to generate the interactive projection. The original illumination parameters are obtained by retrieving the projector operating parameters recorded when generating the interactive projection from the system operation log.

[0107] S63: The refraction range is obtained by comparing the refraction position with the preset mirror specifications.

[0108] The mirror specifications are the geometric parameters such as the size and area of ​​the mirror as set by the technicians.

[0109] The refraction range refers to the area of ​​the drone's mirror that can receive refracted projected light when rotated at various angles at the refraction position. The area of ​​the mirror is calculated by retrieving the various geometric parameters of the mirror, the normal of the projection direction is projected onto the projector, and the parallel plane corresponding to the normal is taken as the standard plane. The area corresponding to each rotation angle range is projected onto the standard plane and the refraction range is defined as the area of ​​each rotation angle range.

[0110] S64: Obtain the original refraction parameters for each refraction range based on the reference variation parameters and the original irradiation parameters.

[0111] The original refraction parameter refers to the portion of the original illumination parameters that, within the allowable variation of the reference variation parameter, can project light into the refraction range. The original illumination parameter that allows the light to intersect the refraction range within the variation of the reference variation parameter is taken as the original refraction parameter.

[0112] S65: Combines the original illumination parameters, original refraction parameters, reference change parameters, and occlusion interaction actions to obtain the calibration tilt angle, and controls the mirror on the UAV to adjust to the calibration tilt angle.

[0113] The calibration tilt angle refers to the angle at which the mirror can recognize occlusion information and assist in light refraction. The calibration tilt angle is obtained by analyzing the original illumination parameters, original refraction parameters, reference change parameters, and occlusion interaction actions, and the mirror on the UAV is then adjusted to this calibration tilt angle. The specific analysis method for the calibration tilt angle is described below.

[0114] Methods for obtaining the calibrated tilt angle include: S70: Obtain constant original parameters by removing the refraction original parameters from the original irradiation parameters.

[0115] Constant original parameters refer to the portion of the original illumination parameters that cannot be varied within the allowable range of the reference variation parameters to project light into the refraction range. The constant original parameters are obtained by removing the refraction original parameters from the original illumination parameters.

[0116] S71: Remove the constant original parameters from the calibration interactive projection and perform projection changes according to the corresponding reference change parameters.

[0117] The residual interactive projection is the partial projection shape corresponding to the original refraction parameters in the calibrated interactive projection. The shape of the remaining interactive projection after removing the constant original parameters from the calibrated interactive projection and performing projection changes with the corresponding reference variation parameters is taken as the residual interactive projection.

[0118] S72: The tilt angle is determined based on the remaining interactive projection, the original refraction parameters, and the occlusion interactive action.

[0119] The calibrated tilt angle was obtained by analyzing the remaining interactive projection, the original refraction parameters, and the occlusion interaction actions. The specific analysis method for calibrating the tilt angle is described below.

[0120] Referring to S15, since the probability of a small portion of the projection is small while the user's actions are being continuously captured, when the calibration tilt angle does not exist, the dual mirror mode and time-division multiplexing mode in S15 are activated. At this time, the specific steps of S72 do not require the use of occlusion interaction actions, and the calibration tilt angle can be obtained only through the remaining interaction projection and refraction original parameters.

[0121] Other methods for obtaining the calibrated tilt angle include: S80: Obtain action recognition feature points based on occlusion interaction actions.

[0122] Action recognition feature points refer to spatial points of key limb parts used to determine the user's interaction intent during occluded interactive actions, such as the center of the palm, fingertips, elbow joint, or shoulder joint. From the skeletal key points of the occluded interactive action, several key points most relevant to the action are selected as action recognition feature points according to the preset action type (such as click, swipe, grab).

[0123] Action type refers to the category of actions set by technicians that enable interaction with the projection.

[0124] S81: The range of action execution is obtained by recognizing action feature points and detecting the recognition trajectory.

[0125] The motion execution range refers to the area covered by the movement of motion recognition feature points in space. It is calculated by connecting the coordinates of the motion recognition feature points in chronological order to form a trajectory, calculating the minimum convex hull or enclosing contour of the trajectory, and then expanding it outward by a preset tolerance radius (e.g., 5 cm). The resulting area is the motion execution range. The tolerance radius is the allowable parameter deviation radius set by the technician for the motion execution range.

[0126] S82: Combine the action execution range with the occlusion range to obtain the marker offset distance.

[0127] The marker offset distance refers to the spatial offset required to ensure that the action execution range and the occlusion range do not overlap (or minimize overlap). It is calculated as the minimum Euclidean distance in the horizontal direction between the action execution range and the occlusion range.

[0128] S83: The refraction range of each mark is defined based on the mark offset distance to obtain the mark refraction range.

[0129] The marked refraction range refers to the range of rotation angles of the mirror corresponding to the action execution range that can reflect the occlusion information, selected from all refraction ranges. By limiting each refraction range, it is determined whether the range refracted by the refraction range meets the marked offset distance to continue displaying the action of the occlusion information. The refraction range that meets this condition is taken as the marked refraction range.

[0130] S84: Based on the overlap between the residual interactive projection and the original refraction parameters, obtain the non-overlapping original calibration parameters and the residual calibration projection.

[0131] The calibration original parameters refer to the refractive original parameters that need to be changed before they can participate in the remaining interactive projection. By comparing the consistency of the ideal light in the remaining interactive projection with the light in the refractive original parameters, the refractive original parameters that are inconsistent with the ideal light are used as the calibration original parameters.

[0132] The calibration residual projection is the portion of the projection that can only be formed in the residual interactive projection after the original calibration parameters have been changed. The projection portion formed by changing the original calibration parameters with the corresponding reference change parameters is used as the calibration residual projection.

[0133] S85: Obtain the deviation variation parameters by calibrating the original parameters, calibrating the remaining projection, and marking the refraction range.

[0134] Deviation variation parameters refer to the amount of change in the original calibration parameters required to make the calibrated residual projection consistent with the ideal residual interactive projection when projecting using one of the marked refraction ranges. This includes the magnitude of changes in the image content, pixel differences, or changes in luminous flux.

[0135] By calibrating the original parameters and marking the refraction range, the geometry of the refracted light spot formed on the projection surface after the projected light is refracted by the mirror is calculated. Based on the calibration of the remaining projection, the refracted light spot is spatially aligned with the calibration of the remaining projection. The minimum adjustment amount (including at least one of the light angle offset, intensity adjustment amount, or position offset) required to make the refracted light spot and the calibration of the remaining projection achieve optimal coincidence is calculated and used as the deviation change parameter.

[0136] S86: Take the marked refraction range corresponding to the minimum deviation change parameter as the target refraction range, and obtain the calibration tilt angle based on the target refraction range.

[0137] The target refraction range refers to the marked refraction range corresponding to the minimum deviation change parameter. The marked refraction range corresponding to the minimum deviation change parameter is taken as the target refraction range, and the angle at which the mirror corresponding to the target refraction range is tilted is taken as the calibration tilt angle.

[0138] Based on the same inventive concept, embodiments of the present invention provide an interactive projection control system based on wireless sensing, comprising: The acquisition module is used to acquire the shooting location and detection image information; A memory for storing the program of the wireless sensing-based interactive projection control method; The processor is used to load and execute programs stored in memory.

[0139] Based on the same inventive concept, embodiments of the present invention provide a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as a wireless sensing-based interactive projection control method.

[0140] Computer storage media include, for example, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media that can store program code.

[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0142] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An interactive projection control method based on wireless sensing, characterized in that, include: Collect the shooting positions of cameras within the venue and the detection image information captured by those cameras; The blind spot is determined based on the detected image information and the preset venue specifications; Identify the interactive users and their movement paths from the detected image information; When the movement path is directed toward the blind spot, a dispatch signal for the drone is generated; In response to scheduling signals, the volume of the blind spot is obtained based on the blind spot and the venue specifications; Based on the volume of the blind spot and the shooting position, the refraction position is obtained, and the drone is controlled to fly to the refraction position to take pictures and obtain supplementary field-of-view images of the blind spot; Interactive images are obtained by supplementing the field of view image, detecting image information, and interacting with the user, and then output as interactive images.

2. The interactive projection control method based on wireless sensing according to claim 1, characterized in that, Methods for obtaining interactive images include: The action and detection information of the person's image are obtained by detecting image information and supplementing visual field images; The detection acceleration and detection inertial force are used to collect detection information; By combining detected acceleration, detected inertial force, and human image motion, the detected and predicted motion is obtained; The orientation of the field of view is determined based on the detected image information and the detection information. The objects observed are determined based on the orientation of the field of view and the specifications of the venue. Interactive images are created by observing objects and detecting and predicting actions.

3. The interactive projection control method based on wireless sensing according to claim 2, characterized in that, Methods for obtaining interactive images include: Based on the test information, cooperation information is obtained; When interactive users exhibit cooperative information, the number of users whose cooperative information is detected is identified from the supplementary visual field image. Retrieve the baseline number of participants from the coordination information; The number of people whose images were blocked was determined by comparing the baseline number of people with the number of people tested. Occlusion information is obtained based on supplementary visual images, the number of people whose vision is obstructed, and cooperation information; The detection acceleration and detection inertial force of occlusion information are used as the occlusion acceleration and occlusion inertial force; Occlusion-based interactive projection is obtained by considering occlusion acceleration, occlusion inertial force, and refraction position. Interactive projection is obtained by combining the detected and predicted actions with the occlusion interactive projection.

4. The interactive projection control method based on wireless sensing according to claim 3, characterized in that, Methods for obtaining occlusion-based interactive projections include: The detection and identification position corresponding to the detected acceleration based on the collected occlusion information; A detection and identification trajectory is formed based on the detection and identification location; Occlusion interaction actions are obtained by detecting and recognizing the trajectory, occlusion acceleration, and occlusion inertial force; The occlusion interaction projection is obtained based on the occlusion interaction action and the refraction position.

5. The interactive projection control method based on wireless sensing according to claim 4, characterized in that, Methods for obtaining occlusion-based interactive projections include: The occlusion range is obtained based on supplementary visual images and coordination information; The occlusion execution range is obtained by combining occlusion information with occlusion interaction actions. The offset distance is obtained by combining the occlusion execution range with the corresponding occlusion range; The offset angle is obtained based on the offset distance and refraction position, and the mirror preset on the drone is controlled to rotate at the offset angle, and the supplementary field of view image is updated; Occlusion interaction projection is obtained based on the updated supplementary field of view image and occlusion interaction actions.

6. The interactive projection control method based on wireless sensing according to claim 5, characterized in that, Methods for obtaining interactive projections include: Update the occlusion interaction action to update the interaction projection, and define the updated interaction projection as the calibration interaction projection; The baseline variation parameters are obtained by comparing the interactive projection with the calibrated interactive projection. Retrieve the original illumination parameters based on the interactive projection; The refraction range is obtained by comparing the refraction position with the preset mirror specifications; The original refraction parameters for each refraction range are obtained based on the reference variation parameters and the original irradiation parameters; The calibration tilt angle is obtained by combining the original illumination parameters, the original refraction parameters, the reference change parameters, and the occlusion interaction actions, and the mirror on the UAV is controlled to adjust the calibration tilt angle.

7. The interactive projection control method based on wireless sensing according to claim 6, characterized in that, Methods for obtaining the calibration tilt angle include: The constant original parameters are obtained by removing the refraction parameters from the original irradiation parameters; The remaining interactive projection is obtained by removing the constant original parameters from the calibration interactive projection and then performing projection changes based on the corresponding reference change parameters. The tilt angle is determined based on the remaining interactive projection, the original refraction parameters, and the occlusion interactive action.

8. The interactive projection control method based on wireless sensing according to claim 7, characterized in that, Other methods for obtaining the calibrated tilt angle include: Action recognition feature points are obtained based on occlusion-interaction actions; The range of action execution is obtained by recognizing feature points and detecting the recognition trajectory. The offset distance is obtained by combining the action execution range with the occlusion range. The refraction range of the marker is obtained by defining each refraction range based on the marker offset distance; Based on the overlap between the residual interactive projection and the original refraction parameters, the non-overlapping original calibration parameters and residual calibration projection are obtained. The deviation variation parameters are obtained by calibrating the original parameters, calibrating the remaining projection, and marking the refraction range. The marked refraction range corresponding to the minimum deviation change parameter is taken as the target refraction range, and the calibrated tilt angle is obtained based on the target refraction range.

9. An interactive projection control system based on wireless sensing, characterized in that, include: The acquisition module is used to acquire the shooting location and detection image information; A memory for storing a program that implements the wireless sensing-based interactive projection control method as described in any one of claims 1 to 8; The processor is used to load and execute programs stored in memory.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 8, which is a wireless sensing-based interactive projection control method.

Images