Equipment control methods, devices, electronic equipment and readable storage media
By using multi-source data fusion and multimodal spatiotemporal graph neural network analysis, the accuracy problem of attitude recognition under occlusion conditions was solved, enabling high-precision intelligent device control and improving user experience.
Patent Information
- Application Number
- CN202510950715.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-10
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-07-10
AI Technical Summary
Existing posture recognition technologies struggle to accurately identify people when they are occluded, especially in multi-sensor fusion methods where they cannot adaptively adjust to dynamic environmental changes, leading to recognition errors.
A multi-source data fusion strategy is adopted to acquire multi-source target object data through posture recognition equipment, calculate the occlusion coefficient and adjust the fusion weight, and combine multimodal spatiotemporal graph neural network to analyze the posture semantic label of the target object and generate a set of control instructions to control the intelligent device.
It improves the accuracy of posture recognition and the precision of intelligent device control, provides high-quality device control services, and enhances the user experience.
Smart Images

Figure CN121011010B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of equipment control technology, and in particular to an equipment control method, an equipment control device, an electronic device, and a computer-readable storage medium. Background Technology
[0002] Posture recognition technology is a technology that identifies and analyzes human postures and movements in real time, and it has broad application prospects in fields such as smart homes.
[0003] In practice, posture recognition technology has undergone several transformations. Initially, simple infrared sensors could only detect the presence of a human body and enable basic equipment control, but could not recognize postures. Then, computer vision technology brought camera-based posture recognition algorithms, which could recognize simple postures. However, due to limitations in computing power and algorithm complexity, they were only suitable for simple, unobstructed environments. When a person was obscured by other people or objects, it was difficult to accurately identify the posture. Summary of the Invention
[0004] In view of the above problems, a device control method, apparatus, electronic device, and readable storage medium are proposed to overcome or at least partially solve the above problems. The specific technical solution is as follows:
[0005] This invention discloses a device control method, in which an attitude recognition device and a smart device are deployed in a designated spatial area. The method includes:
[0006] Acquire first multi-source target object data for the target object; the first multi-source target object data is acquired by at least one of the posture recognition devices;
[0007] The occlusion coefficient of the target object is determined based on the first multi-source target object data corresponding to the target object; the occlusion coefficient is used to characterize the degree to which the target object is occluded when the posture recognition device collects the first multi-source target object data of the target object;
[0008] The first multi-source target object data is fused into a first multimodal feature based on the occlusion coefficient;
[0009] The target pose semantic label of the target object is determined based on the first multimodal feature;
[0010] A control instruction set is generated based on the target posture semantic label, so as to control the corresponding smart device in the specified spatial region according to the control instruction set.
[0011] In one embodiment of the present invention, the attitude recognition device includes at least a camera device, a radar device, and an inertial measurement unit; the first multi-source target object data includes at least visual data collected by the camera device, inertial data collected by the inertial measurement unit, and radar data collected by the radar device.
[0012] In one embodiment of the present invention, determining the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object includes:
[0013] Calculate the structural similarity between the target objects in adjacent visual data based on the visual data;
[0014] Calculate the structural similarity change data between adjacent structural similarities;
[0015] When the structural similarity change data is higher than a preset similarity change threshold, it is determined that the target object is occluded;
[0016] When it is determined that the target object is occluded, an occlusion coefficient is calculated based on the first multi-source target object data.
[0017] In one embodiment of the present invention, determining that the target object is occluded when the structural similarity change data is higher than a preset similarity change threshold includes:
[0018] When the structural similarity change data is higher than a preset similarity change threshold, the first radar data and the second radar data corresponding to the visual data corresponding to the structural similarity change data are acquired; wherein, the time of the first radar data is earlier than the time of the second radar data.
[0019] The first radar point cloud density of the target object is calculated based on the first radar data, and the second radar point cloud density of the target object is calculated based on the second radar data.
[0020] Calculate the point cloud density change data between the first radar point cloud density and the second radar point cloud density;
[0021] When the point cloud density of the first radar is higher than that of the second radar, and the point cloud density change data reaches a preset point cloud density threshold, it is determined that the target object is occluded.
[0022] In one embodiment of the present invention, calculating the occlusion coefficient based on the first multi-source target object data includes:
[0023] occlusion coefficients are calculated based on the structural similarity change data.
[0024] Alternatively, the occlusion coefficient can be calculated based on the point cloud density change data;
[0025] Alternatively, the occlusion coefficient can be calculated based on the structural similarity change data and the point cloud density change data.
[0026] In one embodiment of the present invention, the multimodal spatiotemporal graph neural network includes an input layer, a spatial graph convolutional layer, a temporal convolutional layer, and a modal interaction layer; the step of determining the target pose semantic label of the target object based on the first multimodal feature includes:
[0027] The first multimodal feature is input into a pre-set multimodal spatiotemporal graph neural network to obtain a first pose semantic label of the target object and a first confidence score corresponding to the first pose semantic label; wherein, the input layer is used to receive the first multimodal feature; the spatial graph convolutional layer is used to construct a human joint graph of the target object based on the first multimodal feature, the human joint graph including the spatial relationship between the joints of the human body; the temporal convolutional layer is used to analyze the continuity of the joint movement over time to obtain multi-scale temporal features; the modal interaction layer is used to fuse the first multimodal feature based on the multi-scale temporal features to obtain a first pose semantic label of the target object and a first confidence score corresponding to the first pose semantic label;
[0028] The first pose semantic label corresponding to the highest first confidence score is determined as the target pose semantic label of the target object.
[0029] In one embodiment of the present invention, determining the first pose semantic label corresponding to the highest first confidence score as the target pose semantic label of the target object includes:
[0030] When the highest first confidence score is lower than a preset confidence threshold, collect multi-frame multi-source target object data after the first multi-source target object data as the second multi-source target object data;
[0031] The second multi-source target object data is fused into a second multimodal feature;
[0032] The second multimodal feature is input into a multimodal spatiotemporal graph neural network to obtain the second pose semantic label of the target object and the second confidence score corresponding to the first pose semantic label;
[0033] When the second confidence score is higher than or equal to a preset confidence threshold, the first pose semantic label corresponding to the highest first confidence score is determined as the target pose semantic label of the target object.
[0034] In one embodiment of the present invention, determining the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object includes:
[0035] When there are at least two target objects, the first multi-source target object data corresponding to each target object is determined;
[0036] The occlusion coefficient of each target object in the visual data is determined based on the first multi-source target object data corresponding to each target object.
[0037] In one embodiment of the present invention, determining the first multi-source target object data corresponding to each of the target objects includes:
[0038] The motion characteristics of the target object are obtained based on the radar data;
[0039] The radar data corresponding to each of the target objects is determined based on the motion characteristics.
[0040] A visual skeleton is generated based on the visual data, and the visual data corresponding to each target object is determined based on the spatial distance between the joints of the target objects in the visual skeleton.
[0041] In one embodiment of the present invention, the method further includes:
[0042] When the visual data corresponding to at least two of the target objects matches the radar data of the same target object, the radar data is taken as the target radar data;
[0043] A first motion trend is generated based on the visual data corresponding to each of the target objects, and a second motion trend is generated based on the target radar data;
[0044] The visual data corresponding to the target object that matches the first motion trend and is matched with the second motion trend is used as the visual data for matching the target radar data.
[0045] This invention also discloses a device control apparatus, in which a posture recognition device and an intelligent device are deployed in a designated spatial area, the apparatus comprising:
[0046] The data acquisition module is used to acquire first multi-source target object data for the target object; the first multi-source target object data is acquired by at least one of the posture recognition devices;
[0047] An occlusion coefficient determination module is used to determine the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object; the occlusion coefficient is used to characterize the degree to which the target object is occluded when the pose recognition device collects the first multi-source target object data of the target object;
[0048] The data fusion module is used to fuse the first multi-source target object data into a first multimodal feature based on the occlusion coefficient;
[0049] A pose semantic label determination module is used to determine the target pose semantic label of the target object based on the first multimodal features;
[0050] The device control module is used to generate a set of control instructions based on the target posture semantic tags, so as to control the corresponding smart device in the specified spatial area according to the set of control instructions.
[0051] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0052] The memory is used to store computer programs;
[0053] When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.
[0054] This invention also discloses a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in this invention.
[0055] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.
[0056] The embodiments of the present invention have the following advantages:
[0057] In this embodiment of the invention, a posture recognition device and an intelligent device are deployed in a designated spatial area to acquire first multi-source target object data for a target object. The first multi-source target object data is collected by at least one posture recognition device. Then, the occlusion coefficient of the target object can be determined based on the first multi-source target object data corresponding to the target object. The occlusion coefficient is used to characterize the degree to which the target object is occluded when the posture recognition device collects the first multi-source target object data of the target object. The first multi-source target object data is fused into a first multimodal feature based on the occlusion coefficient, and the target posture semantic label of the target object is determined based on the first multimodal feature. Then, a control instruction set can be generated based on the target posture semantic label, and the corresponding intelligent device in the designated spatial area can be controlled according to the control instruction set. In this embodiment of the invention, when a target object is occluded, an occlusion system can be generated based on first multi-source target object data collected by at least one posture recognition device. Then, based on the occlusion coefficient, the weights of different posture recognition devices used to determine the target posture semantic label of the target object are adjusted. This reduces the proportion of data with a high degree of occlusion of the target object in the first multi-source target object data collected by the posture recognition device when determining the target posture semantic label of the target object, thereby improving the accuracy of target object posture recognition. This allows for accurate identification of the target posture semantic label of the target object, and then accurate invocation of the corresponding smart device based on the target posture semantic label, providing users with high-quality device control services. Attached Figure Description
[0058] Figure 1 This is a flowchart of the steps of a device control method provided in an embodiment of the present invention;
[0059] Figure 2 This is a schematic diagram of a posture recognition process provided in an embodiment of the present invention;
[0060] Figure 3 This is a flowchart of a smart home posture recognition and intelligent control process provided in an embodiment of the present invention under conditions of multiple people's activities and object occlusion;
[0061] Figure 4 This is a structural block diagram of a device control apparatus provided in an embodiment of the present invention;
[0062] Figure 5 This is a schematic diagram of the hardware structure of an electronic device that implements various embodiments of the present invention. Detailed Implementation
[0063] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0064] Reference Figure 1The diagram illustrates a flowchart of a device control method provided in an embodiment of the present invention. An attitude recognition device and a smart device are deployed in a designated spatial area. The method may specifically include the following steps:
[0065] Step 101: Obtain first multi-source target object data for the target object; the first multi-source target object data is obtained by collecting data through at least one of the posture recognition devices.
[0066] For example, the specified spatial area can be a house or other spatial area.
[0067] In a specific implementation, attitude recognition devices and smart devices can be deployed in a designated spatial area. The attitude recognition devices can include at least data acquisition devices such as cameras, radar (e.g., millimeter-wave radar), and inertial measurement units (IMUs). The IMU can be an inertial measurement unit on a device such as a smartwatch worn by the user. Multiple attitude recognition devices can simultaneously collect first multi-source target object data, which can be a human body or a device. This first multi-source target object data can include at least visual data collected by cameras, inertial data (IMU data) collected by the inertial measurement unit, and radar data collected by the radar. The inertial data can include, but is not limited to, acceleration and angular velocity. The smart devices can include lighting equipment, air conditioning equipment, air purifiers, robot vacuums, sterilizers, dehumidifiers, and sensor devices. In some embodiments, the camera and radar devices can be placed in the same location.
[0068] Step 102: Determine the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object; the occlusion coefficient is used to characterize the degree to which the target object is occluded when the posture recognition device collects the first multi-source target object data of the target object.
[0069] In practical implementation, when the target object moves in a specified spatial area, multiple posture recognition devices will follow and collect the first multi-source target object data of the target object. However, sometimes the posture recognition devices may be blocked by objects such as furniture, resulting in blurred images in the collected visual data or large-area occlusion of the target object in the collected visual data (e.g., 70% occlusion). For example, the camera device will have a blurred image because the lower half of the human body is blocked by the TV cabinet. At this time, the degree of occlusion of the target object can be dynamically evaluated based on the first multi-source target object data corresponding to the target object and an occlusion coefficient can be generated. The occlusion coefficient is used to characterize the degree of occlusion of the target object when the posture recognition device collects the first multi-source target object data of the target object.
[0070] Step 103: Fuse the first multi-source target object data into a first multimodal feature based on the occlusion coefficient.
[0071] In this embodiment of the invention, the degree of occlusion of the target object can be dynamically evaluated and an occlusion coefficient can be generated based on the first multi-source target object data corresponding to the collected target object. Different fusion weights can be assigned to the first multi-source target object data collected by different pose recognition devices based on the occlusion coefficient. Then, the first multi-source target object data can be fused into a first multimodal feature according to the fusion weight.
[0072] For example, when the target object is lightly occluded or not occluded at all, visual data is used as the primary source, and visual data and radar data are aligned and stitched together to obtain the first multimodal feature. When the target object is heavily occluded, radar data is used as the primary source, and visual data and radar data are aligned and stitched together to obtain the first multimodal feature. Of course, if there are other first multi-source target object data besides visual data and radar data, such as inertial data, these can also be aligned and stitched together with visual data and radar data to obtain the first multimodal feature.
[0073] Step 104: Determine the target pose semantic label of the target object based on the first multimodal feature.
[0074] Step 105: Generate a control instruction set based on the target posture semantic tag, and control the corresponding smart device in the specified spatial region according to the control instruction set.
[0075] For example, assuming the target object is a human body, the gesture semantic label can include "bending over", "raising hand", "walking towards the door", etc.
[0076] In this embodiment of the invention, the target pose semantic label of the target object can be determined by analyzing the first multimodal features. Simultaneously, the environmental data of the target object can be determined by combining the first multi-source target object data or data collected by other devices. For example, if the target object is next to a cabinet, the target pose semantic label and environmental data can be input into a reinforcement learning decision engine to generate a corresponding control instruction set. This control instruction set can then be used to invoke the corresponding smart device in the specified spatial area. For instance, assuming the target pose semantic label is "bending over" and the environmental data is "the target object is next to a cabinet," it can be determined that the target object is looking for something next to the cabinet. If there is a lighting device next to the target object that is not turned on, a control instruction set to turn on the lighting device can be generated to control the lighting device to turn on.
[0077] In the above-described device control method, a posture recognition device and an intelligent device are deployed in a designated spatial area to acquire first multi-source target object data for a target object. The first multi-source target object data is acquired by at least one posture recognition device. Then, the occlusion coefficient of the target object can be determined based on the first multi-source target object data corresponding to the target object. The occlusion coefficient is used to characterize the degree to which the target object is occluded when the posture recognition device acquires the first multi-source target object data of the target object. The first multi-source target object data is fused into a first multimodal feature based on the occlusion coefficient, and the target posture semantic label of the target object is determined based on the first multimodal feature. Then, a control instruction set can be generated based on the target posture semantic label, and the corresponding intelligent device in the designated spatial area can be controlled according to the control instruction set. In this embodiment of the invention, when a target object is occluded, an occlusion system can be generated based on first multi-source target object data collected by at least one posture recognition device. Then, based on the occlusion coefficient, the weights of different posture recognition devices used to determine the target posture semantic label of the target object are adjusted. This reduces the proportion of data with a high degree of occlusion of the target object in the first multi-source target object data collected by the posture recognition device when determining the target posture semantic label of the target object, thereby improving the accuracy of target object posture recognition. This allows for accurate identification of the target posture semantic label of the target object, and then accurate invocation of the corresponding smart device based on the target posture semantic label, providing users with high-quality device control services.
[0078] In one embodiment of the present invention, step 102, determining the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object, may include:
[0079] Calculate the structural similarity between the target objects in adjacent visual data based on the visual data;
[0080] Calculate the structural similarity change data between adjacent structural similarities;
[0081] When the structural similarity change data is higher than a preset similarity change threshold, it is determined that the target object is occluded;
[0082] When it is determined that the target object is occluded, an occlusion coefficient is calculated based on the first multi-source target object data.
[0083] Structural Similarity (SSIM) is a metric that measures the similarity of two images (usually consecutive images) in terms of structural information.
[0084] In this embodiment of the invention, the structural similarity between target objects in adjacent visual data (images) can be calculated based on visual data, and the structural similarity change data between the calculated adjacent structural similarities can be determined. When the structural similarity change data is higher than a preset similarity change threshold, it indicates that the target object is occluded. At this time, an occlusion coefficient can be calculated based on the first multi-source target object data to adaptively adjust the weights of the visual data and radar data used to generate the first multimodal features, thereby improving the target pose semantic label generated based on the first multimodal features. This allows for accurate control of the corresponding smart device to provide device services to the user, improving the user experience.
[0085] In one embodiment of the present invention, determining that the target object is occluded when the structural similarity change data is higher than a preset similarity change threshold may include:
[0086] When the structural similarity change data is higher than a preset similarity change threshold, the first radar data and the second radar data corresponding to the visual data corresponding to the structural similarity change data are acquired; wherein, the time of the first radar data is earlier than the time of the second radar data.
[0087] The first radar point cloud density of the target object is calculated based on the first radar data, and the second radar point cloud density of the target object is calculated based on the second radar data.
[0088] Calculate the point cloud density change data between the first radar point cloud density and the second radar point cloud density;
[0089] When the point cloud density of the first radar is higher than that of the second radar, and the point cloud density change data reaches a preset point cloud density threshold, it is determined that the target object is occluded.
[0090] In practical implementation, when the structural similarity change data exceeds a preset similarity change threshold, it indicates that the target object is occluded, such as by furniture. However, it could also be a misjudgment due to the target object's color being similar to its surrounding environment. To accurately determine whether the target object is occluded, when the structural similarity change data exceeds the preset similarity change threshold, radar data corresponding to the visual data corresponding to the structural similarity change data can be obtained. The radar data can include first radar data and second radar data, with the first radar data being earlier than the second radar data. Then, the first radar point cloud density can be calculated based on the first radar data, and the second radar point cloud density can be calculated based on the second radar data. Radar point cloud density is a quantitative indicator describing the number of effective radar reflection points per unit area, reflecting the completeness of radar detection of the surface features of the detected object (e.g., the target object).
[0091] The point cloud density change data between the first and second radar point cloud densities of the target object is calculated. When the first radar point cloud density of the target object is higher than the second radar point cloud density, and the point cloud density change data reaches a preset point cloud density threshold, it indicates that the effective point cloud of the target object that the radar can detect has decreased, for example, from 100% to 50%, exceeding the preset point cloud density threshold. This indicates that the target object in the visual data is occluded. By combining visual data and radar data to determine whether the target object is occluded, misjudgments caused by relying solely on visual data can be avoided. Next, an occlusion coefficient can be calculated based on the first multi-source target object data to adaptively adjust the weights of the visual data and radar data used to generate the first multimodal features. This improves the target pose semantic label generated based on the first multimodal features, thereby accurately controlling the corresponding smart device to provide device services to the user and improving the user experience.
[0092] In one embodiment of the present invention, calculating the occlusion coefficient based on the first multi-source target object data may include:
[0093] occlusion coefficients are calculated based on the structural similarity change data.
[0094] Alternatively, the occlusion coefficient can be calculated based on the point cloud density change data;
[0095] Alternatively, the occlusion coefficient can be calculated based on the structural similarity change data and the point cloud density change data.
[0096] In this embodiment of the invention, when calculating the occlusion coefficient, the occlusion coefficient can be calculated based on the structural similarity change data, or it can be calculated based on the structural similarity change data and the point cloud density change data. This embodiment of the invention does not impose any restrictions on this. The corresponding calculation method of the occlusion coefficient can be selected according to actual needs, such as different scenarios or accuracy requirements, thus realizing diversified calculation of the occlusion coefficient.
[0097] In one embodiment of the present invention, the multimodal spatiotemporal graph neural network may include an input layer, a spatial graph convolutional layer, a temporal convolutional layer, and a modal interaction layer; the step of determining the target pose semantic label of the target object based on the first multimodal feature may include:
[0098] The first multimodal feature is input into a pre-set multimodal spatiotemporal graph neural network to obtain a first pose semantic label of the target object and a first confidence score corresponding to the first pose semantic label; wherein, the input layer is used to receive the first multimodal feature; the spatial graph convolutional layer is used to construct a human joint graph of the target object based on the first multimodal feature, the human joint graph including the spatial relationship between the joints of the human body; the temporal convolutional layer is used to analyze the continuity of the joint movement over time to obtain multi-scale temporal features; the modal interaction layer is used to fuse the first multimodal feature based on the multi-scale temporal features to obtain a first pose semantic label of the target object and a first confidence score corresponding to the first pose semantic label;
[0099] The first pose semantic label corresponding to the highest first confidence score is determined as the target pose semantic label of the target object.
[0100] In this embodiment of the invention, the multimodal spatiotemporal graph neural network can extract complex spatiotemporal dependencies by integrating graph neural networks and various temporal learning methods. (Refer to...) Figure 2 This is a schematic diagram of a posture recognition process provided in an embodiment of the present invention, including a multimodal spatiotemporal graph neural network. Specifically, the multimodal spatiotemporal graph neural network may include an input layer, a spatial graph convolutional layer, a temporal convolutional layer, and a modal interaction layer. The input layer has a four-dimensional tensor structure with batch size, time step, number of joints, and feature dimensions, including position, acceleration, velocity, and point cloud reflection intensity. The spatial convolutional layer constructs a dynamic graph with 17 human joints and performs multi-scale feature extraction. The temporal convolutional layer uses dilated causal convolution to capture short, medium, and long-term action patterns. The modal interaction layer includes cross-modal attention gating and a competition-cooperation mechanism. The competition-cooperation mechanism mainly calculates the cosine similarity of the motion vector fields of different individual joints to separate human interaction actions, while cooperation involves weighted averaging of multiple sensor features of the same target object. In some embodiments, the multimodal spatiotemporal graph neural network can use Monte Carlo variance to calculate posture semantic labels and confidence scores.
[0101] Specifically, the first multimodal feature is input into the multimodal spatiotemporal graph neural network. After processing by the input layer, a human joint graph is first constructed through a spatial graph convolutional layer to extract the spatial relationships between joints. Then, a temporal convolutional layer is used to capture multi-scale temporal features and analyze the continuity of actions. Finally, through the cross-modal attention gating mechanism of the modal interaction layer, multi-source target object data such as visual data, radar data, and IMU data are selectively fused to obtain the first pose semantic label of the target object and the first confidence score corresponding to the first pose semantic label. Then, the first pose semantic label corresponding to the highest first confidence score is usually determined as the target pose semantic label of the target object. By using the multimodal spatiotemporal graph neural network, multi-source target object data such as visual data, radar data, and IMU data can be accurately fused to obtain accurate pose semantic labels and confidence scores, which can improve the accuracy of target object pose recognition.
[0102] In one embodiment of the present invention, determining the first pose semantic label corresponding to the highest first confidence score as the target pose semantic label of the target object includes:
[0103] When the highest first confidence score is lower than a preset confidence threshold, collect multi-frame multi-source target object data after the first multi-source target object data as the second multi-source target object data;
[0104] The second multi-source target object data is fused into a second multimodal feature;
[0105] The second multimodal feature is input into a multimodal spatiotemporal graph neural network to obtain the second pose semantic label of the target object and the second confidence score corresponding to the first pose semantic label;
[0106] When the second confidence score is higher than or equal to a preset confidence threshold, the first pose semantic label corresponding to the highest first confidence score is determined as the target pose semantic label of the target object.
[0107] If the highest initial confidence score output by the multimodal spatiotemporal graph neural network is lower than the preset confidence threshold, it indicates that the multimodal spatiotemporal graph neural network may have a calculation error, or that the data input to the multimodal spatiotemporal graph neural network is inaccurate due to factors such as noise and occlusion in a complex environment. In this case, secondary verification is required. (Refer to...) Figure 2The multimodal spatiotemporal graph neural network (MPNN) or the data input to it can be validated using uncertainty quantification. Uncertainty quantification refers to validation through mathematical and statistical methods. Specifically, the secondary validation process first collects multiple frames of multimodal target object data following the first multimodal target object data as the second multimodal target object data. For example, it could be 3-5 frames of multimodal target object data. Then, the second multimodal target object data is fused into a second multimodal feature. The generation process of the second multimodal feature can refer to the generation process of the first multimodal feature, and will not be elaborated further.
[0108] Then, the second multimodal feature is input into the multimodal spatiotemporal graph neural network to obtain the second pose semantic label of the target object and the second confidence score corresponding to the first pose semantic label. When the second confidence score is higher than or equal to the preset confidence threshold, it indicates that the data output by the multimodal spatiotemporal graph neural network is reliable. Then, the first pose semantic label corresponding to the highest first confidence score can be determined as the target pose semantic label of the target object. Based on the target pose semantic label, the corresponding smart device is controlled, thereby ensuring that the smart device will not be erroneously controlled and ensuring the user experience.
[0109] In one embodiment of the present invention, determining the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object may include:
[0110] When there are at least two target objects, the first multi-source target object data corresponding to each target object is determined;
[0111] The occlusion coefficient of each target object in the visual data is determined based on the first multi-source target object data corresponding to each target object.
[0112] In the specific implementation, the target object in the specified spatial region can be one or multiple, as shown in the reference. Figure 2 When there are multiple target objects, such as more than two individuals, it is a multi-person scene. For multi-person scenes, pose decoupling is required. Individual separation algorithms such as competition-cooperative attention mechanism can be used to separate individuals. The occlusion coefficient of each target object in the visual data is determined for the first multi-source target object data corresponding to each target object. Of course, the first multimodal feature input to the multimodal spatiotemporal graph neural network is also the first multimodal feature corresponding to each individual. Thus, the multimodal spatiotemporal graph neural network can output the pose semantic label and confidence score corresponding to each individual.
[0113] In one embodiment of the present invention, determining the first multi-source target object data corresponding to each of the target objects may include:
[0114] The motion characteristics of the target object are obtained based on the radar data;
[0115] The radar data corresponding to each of the target objects is determined based on the motion characteristics.
[0116] A visual skeleton is generated based on the visual data, and the visual data corresponding to each target object is determined based on the spatial distance between the joints of the target objects in the visual skeleton.
[0117] In this embodiment of the invention, radar data is processed by radar Doppler velocity clustering to obtain the motion characteristics of the target object. Then, radar data with motion characteristics can be divided into radar data of the same target object, thereby initially distinguishing different human bodies.
[0118] Then, a visual skeleton can be generated based on the visual data using image processing techniques. The visual skeleton can include the positions of the joints of an individual. Then, based on the spatial distance between the joints of the target object in the visual skeleton, it can be determined whether the corresponding visual data belong to the same individual. For example, if the spatial distance between the joints in the visual skeleton is relatively large, it can be determined that the corresponding visual data do not belong to the same individual.
[0119] By using radar Doppler velocity clustering and image processing technology, radar data and visual data corresponding to each target object in radar data and visual data can be accurately separated. This avoids the inability of multimodal spatiotemporal graph neural networks to output accurate attitude semantic labels and confidence scores due to misidentification of target objects, thus ensuring that intelligent devices do not make incorrect adjustments and guaranteeing user experience.
[0120] In one embodiment of the present invention, the method may further include:
[0121] When the visual data corresponding to at least two of the target objects matches the radar data of the same target object, the radar data is taken as the target radar data;
[0122] A first motion trend is generated based on the visual data corresponding to each of the target objects, and a second motion trend is generated based on the target radar data;
[0123] The visual data corresponding to the target object that matches the first motion trend and is matched with the second motion trend is used as the visual data for matching the target radar data.
[0124] In practical implementation, radar Doppler velocity clustering and image processing techniques can accurately separate the radar data and visual data corresponding to each target object in the radar data and visual data. However, overlapping areas may exist. For example, the visual data corresponding to at least two target objects may match the radar data of the same target object. In this case, the Hungarian algorithm can be used to generate a first motion trend based on the visual data corresponding to each target object, and a second motion trend based on the target radar data. The motion trend can include the target object's motion direction, velocity change, acceleration pattern, etc. Then, the visual data corresponding to the target object that matches the first motion trend of the second motion trend can be used. For example, assuming there are individuals 1 and 2, if the first motion trend of individual 2's hand joint matches the second motion trend of individual 2's hand joint, then individual 2 can be identified as the visual data matching the individual in the target radar data. Subsequently, interactive action recognition can be performed based on the visual data and radar data corresponding to each individual. Specifically, by analyzing the relative motion trends between individuals, such as determining whether they are shaking hands or passing by, the individual's posture can be analyzed more accurately, thereby better understanding the user's intention.
[0125] By calculating the first motion trend of each individual in the visual data and the second motion trend of each individual in the radar data, and matching the second motion trend based on the first motion trend, it is possible to accurately separate target objects with overlapping areas, and ensure that the posture of each target object can be accurately identified so as to accurately control the intelligent device.
[0126] Existing posture recognition technologies, such as those used in smart homes, are mainly divided into two categories: those based on single sensors and those based on multi-sensor fusion. Single-sensor technologies rely solely on camera equipment for posture recognition, which is prone to errors due to human occlusion. Multi-sensor fusion technology improves performance to some extent by fusing camera data with millimeter-wave radar data. However, existing fusion methods are mostly simple data stitching or fixed-weight fusion, which cannot adaptively adjust to dynamic changes in the environment.
[0127] The embodiments of the present invention improve the accuracy of posture recognition and the level of intelligence of device control through innovative sensor fusion strategies, optimized deep learning models and intelligent control decision-making mechanisms, thereby providing users with a more comfortable, portable and intelligent living environment.
[0128] To enable those skilled in the art to better understand the embodiments of the present invention, a specific example is provided below. (Refer to...) Figure 3 This is a flowchart illustrating a smart home posture recognition and intelligent control process in an environment with multiple people and object occlusion, provided in this embodiment of the invention. The specific steps include:
[0129] S31: Synchronous acquisition of multi-source data (multi-source target object data). This is achieved through camera equipment and millimeter-wave radar; if the user has a smartwatch or other device, IMU data, such as acceleration and angular velocity, can also be acquired.
[0130] S32: Dynamic Feature Fusion. By calculating the structural similarity of visual data and the radar point cloud density of radar data, the degree of user occlusion is dynamically assessed, and fusion weights for visual and radar data are assigned based on an occlusion coefficient formula. When the occlusion degree is low, visual data takes precedence; when the occlusion degree is severe, radar data takes precedence. Simultaneously, RGB images, depth maps, radar point cloud skeletons, and IMU data are aligned and stitched together to fuse into multimodal features.
[0131] S33: Pose Recognition Inference; Multimodal features are input into a multimodal spatiotemporal graph neural network. First, a human joint graph is constructed through spatial graph convolutional layers to extract spatial relationships between joints. Then, temporal convolutional layers are used to capture multi-scale temporal features and analyze the continuity of actions. Finally, a cross-modal attention gating mechanism is used to selectively fuse visual, radar, and IMU features. For multi-person scenarios, a competitive-cooperative attention mechanism is used to separate individuals and output the pose semantic label and confidence score of each individual. A secondary verification process is initiated for low confidence scores.
[0132] S34: Intelligent decision execution; inputting multi-dimensional information such as attitude recognition labels and environmental data output by the multimodal spatiotemporal graph neural network into the reinforcement learning decision engine to generate the optimal control instruction set to control the corresponding intelligent device.
[0133] S35: Online learning updates. Specifically, data can be continuously collected to update the multimodal spatiotemporal graph neural network to ensure the accuracy of the semantic pose labels and confidence scores output by the multimodal spatiotemporal graph neural network.
[0134] For example, when a user is looking for the remote control in the living room, they are suddenly blocked by the TV cabinet. The camera equipment detects that the user's lower body is obstructed, and the obstruction area is relatively large, that is, the degree of obstruction is high. The system switches to radar-dominated mode, uses radar point cloud to infer the position of the user's legs, and the multimodal spatiotemporal neural network detects that the user's upper body is leaning forward and the legs are squatting. It is determined that the user is bending over to find something, and the system turns on the lights near the user to help the user find the thing.
[0135] In summary, the embodiments of this invention propose a dynamic multimodal fusion mechanism based on occlusion perception, which achieves adaptive weight allocation of sensors (such as camera devices and radar devices) through an interpretable mathematical model; designs a multimodal spatiotemporal graph neural network, adopting a hierarchical graph convolutional structure and a competitive-cooperative attention mechanism to effectively solve the pose ambiguity problem in multi-person interaction scenarios; and constructs a reinforcement learning-driven intelligent decision-making system, integrating inverse reinforcement learning and safety constraint mechanisms to achieve personalized, safe, and reliable intelligent device control, providing users with high-quality intelligent device services.
[0136] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.
[0137] Reference Figure 4 The diagram illustrates a structural block diagram of a device control apparatus provided in an embodiment of the present invention. An attitude recognition device and a smart device are deployed in a designated spatial area. The apparatus may specifically include the following modules:
[0138] Data acquisition module 401 is used to acquire first multi-source target object data for the target object; the first multi-source target object data is acquired by at least one of the posture recognition devices;
[0139] The occlusion coefficient determination module 402 is used to determine the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object; the occlusion coefficient is used to characterize the degree to which the target object is occluded when the posture recognition device collects the first multi-source target object data of the target object;
[0140] Data fusion module 403 is used to fuse the first multi-source target object data into a first multimodal feature according to the occlusion coefficient;
[0141] The pose semantic label determination module 404 is used to determine the target pose semantic label of the target object based on the first multimodal features;
[0142] The device control module 405 is used to generate a set of control instructions based on the target posture semantic tags, so as to control the corresponding smart device in the specified spatial area according to the set of control instructions.
[0143] In one embodiment of the present invention, the attitude recognition device includes at least a camera device, a radar device, and an inertial measurement unit; the first multi-source target object data includes at least visual data collected by the camera device, inertial data collected by the inertial measurement unit, and radar data collected by the radar device.
[0144] In one embodiment of the present invention, the occlusion coefficient determination module 402 is used for:
[0145] Calculate the structural similarity between the target objects in adjacent visual data based on the visual data;
[0146] Calculate the structural similarity change data between adjacent structural similarities;
[0147] When the structural similarity change data is higher than a preset similarity change threshold, it is determined that the target object is occluded;
[0148] When it is determined that the target object is occluded, an occlusion coefficient is calculated based on the first multi-source target object data.
[0149] In one embodiment of the present invention, the occlusion coefficient determination module 402 is used for:
[0150] When the structural similarity change data is higher than a preset similarity change threshold, the first radar data and the second radar data corresponding to the visual data corresponding to the structural similarity change data are acquired; wherein, the time of the first radar data is earlier than the time of the second radar data.
[0151] The first radar point cloud density of the target object is calculated based on the first radar data, and the second radar point cloud density of the target object is calculated based on the second radar data.
[0152] Calculate the point cloud density change data between the first radar point cloud density and the second radar point cloud density;
[0153] When the point cloud density of the first radar is higher than that of the second radar, and the point cloud density change data reaches a preset point cloud density threshold, it is determined that the target object is occluded.
[0154] In one embodiment of the present invention, the occlusion coefficient determination module 402 is used for:
[0155] occlusion coefficients are calculated based on the structural similarity change data.
[0156] Alternatively, the occlusion coefficient can be calculated based on the point cloud density change data;
[0157] Alternatively, the occlusion coefficient can be calculated based on the structural similarity change data and the point cloud density change data.
[0158] In one embodiment of the present invention, the multimodal spatiotemporal graph neural network includes an input layer, a spatial graph convolutional layer, a temporal convolutional layer, and a modal interaction layer; the pose semantic label determination module 404 is used for:
[0159] The first multimodal feature is input into a pre-set multimodal spatiotemporal graph neural network to obtain a first pose semantic label of the target object and a first confidence score corresponding to the first pose semantic label; wherein, the input layer is used to receive the first multimodal feature; the spatial graph convolutional layer is used to construct a human joint graph of the target object based on the first multimodal feature, the human joint graph including the spatial relationship between the joints of the human body; the temporal convolutional layer is used to analyze the continuity of the joint movement over time to obtain multi-scale temporal features; the modal interaction layer is used to fuse the first multimodal feature based on the multi-scale temporal features to obtain a first pose semantic label of the target object and a first confidence score corresponding to the first pose semantic label;
[0160] The first pose semantic label corresponding to the highest first confidence score is determined as the target pose semantic label of the target object.
[0161] In one embodiment of the present invention, the pose semantic label determination module 404 is used for:
[0162] When the highest first confidence score is lower than a preset confidence threshold, collect multi-frame multi-source target object data after the first multi-source target object data as the second multi-source target object data;
[0163] The second multi-source target object data is fused into a second multimodal feature;
[0164] The second multimodal feature is input into a multimodal spatiotemporal graph neural network to obtain the second pose semantic label of the target object and the second confidence score corresponding to the first pose semantic label;
[0165] When the second confidence score is higher than or equal to a preset confidence threshold, the first pose semantic label corresponding to the highest first confidence score is determined as the target pose semantic label of the target object.
[0166] In one embodiment of the present invention, the occlusion coefficient determination module 402 is used for:
[0167] When there are at least two target objects, the first multi-source target object data corresponding to each target object is determined;
[0168] The occlusion coefficient of each target object in the visual data is determined based on the first multi-source target object data corresponding to each target object.
[0169] In one embodiment of the present invention, the occlusion coefficient determination module 402 is used for:
[0170] The motion characteristics of the target object are obtained based on the radar data;
[0171] The radar data corresponding to each of the target objects is determined based on the motion characteristics.
[0172] A visual skeleton is generated based on the visual data, and the visual data corresponding to each target object is determined based on the spatial distance between the joints of the target objects in the visual skeleton.
[0173] In one embodiment of the present invention, the device further includes: a matching module, configured to:
[0174] When the visual data corresponding to at least two of the target objects matches the radar data of the same target object, the radar data is taken as the target radar data;
[0175] A first motion trend is generated based on the visual data corresponding to each of the target objects, and a second motion trend is generated based on the target radar data;
[0176] The visual data corresponding to the target object that matches the first motion trend and is matched with the second motion trend is used as the visual data for matching the target radar data.
[0177] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.
[0178] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described device control method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.
[0179] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described device control method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0180] This invention also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described device control method embodiments, and can achieve the same technical effects. To avoid repetition, it will not be described again here.
[0181] Figure 5 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.
[0182] The electronic device 500 includes, but is not limited to, components such as: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510, and a power supply 511. Those skilled in the art will understand that... Figure 5 The electronic device structures shown are not intended to limit the electronic device. An electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In embodiments of the present invention, the electronic device includes, but is not limited to, mobile phones, tablet computers, laptops, PDAs, in-vehicle terminals, wearable devices, and pedometers.
[0183] It should be understood that, in this embodiment of the invention, the radio frequency unit 501 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 510; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 501 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc. Furthermore, the radio frequency unit 501 can also communicate with networks and other devices through a wireless communication system.
[0184] The electronic device provides users with wireless broadband internet access through the network module 502, such as helping users send and receive emails, browse web pages, and access streaming media.
[0185] The audio output unit 503 can convert audio data received by the radio frequency unit 501 or the network module 502 or stored in the memory 509 into audio signals and output them as sound. Furthermore, the audio output unit 503 can also provide audio output related to specific functions performed by the electronic device 500 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 503 includes a speaker, a buzzer, and a receiver, etc.
[0186] Input unit 504 is used to receive audio or video signals. Input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on display unit 506. The image frames processed by GPU 5041 can be stored in memory 509 (or other storage medium) or transmitted via radio frequency unit 501 or network module 502. Microphone 5042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 501 in telephone call mode.
[0187] The electronic device 500 also includes at least one sensor 505, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 5061 according to the ambient light level, and the proximity sensor can turn off the display panel 5061 and / or backlight when the electronic device 500 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 505 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.
[0188] The display unit 506 is used to display information input by the user or information provided to the user. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.
[0189] User input unit 507 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 507 includes a touch panel 5071 and other input devices 5072. Touch panel 5071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 5071). Touch panel 5071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 510, which receives and executes commands from the processor 510. In addition, touch panel 5071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides touch panel 5071, user input unit 507 may also include other input devices 5072. Specifically, other input devices 5072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.
[0190] Furthermore, the touch panel 5071 can cover the display panel 5061. When the touch panel 5071 detects a touch operation on or near it, it transmits the information to the processor 510 to determine the type of touch event. Subsequently, the processor 510 provides corresponding visual output on the display panel 5061 based on the type of touch event. Although in Figure 5 In this embodiment, the touch panel 5071 and the display panel 5061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 5071 and the display panel 5061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.
[0191] Interface unit 508 serves as an interface for connecting external devices to electronic device 500. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 508 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 500, or it can be used to transmit data between electronic device 500 and external devices.
[0192] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 509 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0193] The processor 510 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 509, and by calling data stored in the memory 509, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 510 may include one or more processing units; preferably, the processor 510 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 510.
[0194] The electronic device 500 may also include a power supply 511 (such as a battery) for supplying power to various components. Preferably, the power supply 511 can be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.
[0195] In addition, the electronic device 500 includes some functional modules not shown, which will not be described in detail here.
[0196] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0198] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.
[0199] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0200] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0201] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0202] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0203] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0204] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0205] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A device control method, characterized in that, The method involves deploying a posture recognition device and a smart device in a designated spatial area, and includes: Acquire first multi-source target object data for the target object; the first multi-source target object data is acquired by at least one of the posture recognition devices; The occlusion coefficient of the target object is determined based on the first multi-source target object data corresponding to the target object; the occlusion coefficient is used to characterize the degree to which the target object is occluded when the posture recognition device collects the first multi-source target object data of the target object; The first multi-source target object data is fused into a first multimodal feature based on the occlusion coefficient; The target pose semantic label of the target object is determined based on the first multimodal feature; A control instruction set is generated based on the target posture semantic tag, so as to control the corresponding smart device in the specified spatial area according to the control instruction set; Determining the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object includes: Calculate the structural similarity between the target objects in adjacent visual data based on the visual data; Calculate the change in structural similarity between adjacent structural similarities; When the structural similarity change data is higher than a preset similarity change threshold, it is determined that the target object is occluded; When it is determined that the target object is occluded, an occlusion coefficient is calculated based on the first multi-source target object data; Determining that the target object is occluded when the structural similarity change data is higher than a preset similarity change threshold includes: When the structural similarity change data is higher than a preset similarity change threshold, the first radar data and the second radar data corresponding to the visual data corresponding to the structural similarity change data are acquired; wherein, the time of the first radar data is earlier than the time of the second radar data. The first radar point cloud density of the target object is calculated based on the first radar data, and the second radar point cloud density of the target object is calculated based on the second radar data. Calculate the point cloud density change data between the first radar point cloud density and the second radar point cloud density; When the point cloud density of the first radar is higher than that of the second radar, and the point cloud density change data reaches a preset point cloud density threshold, it is determined that the target object is occluded. The occlusion coefficient is calculated based on the first multi-source target object data, including: occlusion coefficients are calculated based on the structural similarity change data. Alternatively, the occlusion coefficient can be calculated based on the point cloud density change data; Alternatively, the occlusion coefficient can be calculated based on the structural similarity change data and the point cloud density change data.
2. The method according to claim 1, characterized in that, The attitude recognition device includes at least a camera device, a radar device, and an inertial measurement unit; the first multi-source target object data includes at least visual data collected by the camera device, inertial data collected by the inertial measurement unit, and radar data collected by the radar device.
3. The method according to claim 1, characterized in that, Determining the target pose semantic label of the target object based on the first multimodal feature includes: The first multimodal feature is input into a pre-set multimodal spatiotemporal graph neural network to obtain a first pose semantic label of the target object and a first confidence score corresponding to the first pose semantic label. The multimodal spatiotemporal graph neural network includes an input layer, a spatial graph convolutional layer, a temporal convolutional layer, and a modal interaction layer. The input layer receives the first multimodal feature. The spatial graph convolutional layer constructs a human joint graph of the target object based on the first multimodal feature, the human joint graph including the spatial relationships between the joints of the human body. The temporal convolutional layer analyzes the continuity of the joint movements over time to obtain multi-scale temporal features. The modal interaction layer fuses the first multimodal feature with the multi-scale temporal features to obtain a first pose semantic label of the target object and a first confidence score corresponding to the first pose semantic label. The first pose semantic label corresponding to the highest first confidence score is determined as the target pose semantic label of the target object.
4. The method according to claim 3, characterized in that, The step of determining the first pose semantic label corresponding to the highest first confidence score as the target pose semantic label of the target object includes: When the highest first confidence score is lower than a preset confidence threshold, collect multi-frame multi-source target object data after the first multi-source target object data as the second multi-source target object data; The second multi-source target object data is fused into a second multimodal feature; The second multimodal feature is input into a multimodal spatiotemporal graph neural network to obtain the second pose semantic label of the target object and the second confidence score corresponding to the first pose semantic label; When the second confidence score is higher than or equal to a preset confidence threshold, the first pose semantic label corresponding to the highest first confidence score is determined as the target pose semantic label of the target object.
5. The method according to claim 2, characterized in that, Determining the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object includes: When there are at least two target objects, the first multi-source target object data corresponding to each target object is determined; The occlusion coefficient of each target object in the visual data is determined based on the first multi-source target object data corresponding to each target object.
6. The method according to claim 5, characterized in that, Determining the first multi-source target object data corresponding to each of the target objects includes: The motion characteristics of the target object are obtained based on the radar data; The radar data corresponding to each of the target objects is determined based on the motion characteristics. A visual skeleton is generated based on the visual data, and the visual data corresponding to each target object is determined based on the spatial distance between the joints of the target objects in the visual skeleton.
7. The method according to claim 6, characterized in that, The method further includes: When the visual data corresponding to at least two of the target objects matches the radar data of the same target object, the radar data is taken as the target radar data; A first motion trend is generated based on the visual data corresponding to each of the target objects, and a second motion trend is generated based on the target radar data; The visual data corresponding to the target object that matches the first motion trend and is matched with the second motion trend is used as the visual data for matching the target radar data.
8. A device control apparatus, characterized in that, A gesture recognition device and an intelligent device are deployed in a designated spatial area, the device comprising: The data acquisition module is used to acquire first multi-source target object data for the target object; the first multi-source target object data is acquired by at least one of the posture recognition devices; An occlusion coefficient determination module is used to determine the occlusion coefficient of the target object based on the first multi-source target object data corresponding to the target object; The data fusion module is used to fuse the first multi-source target object data into a first multimodal feature based on the occlusion coefficient; A pose semantic label determination module is used to determine the target pose semantic label of the target object based on the first multimodal features; The device control module is used to generate a set of control instructions based on the target posture semantic tags, so as to control the corresponding smart device in the specified spatial area according to the set of control instructions; The occlusion coefficient determination module is further configured to: Calculate the structural similarity between the target objects in adjacent visual data based on the visual data; Calculate the change in structural similarity between adjacent structural similarities; When the structural similarity change data is higher than a preset similarity change threshold, it is determined that the target object is occluded; When it is determined that the target object is occluded, an occlusion coefficient is calculated based on the first multi-source target object data; The occlusion coefficient determination module is also used for: When the structural similarity change data is higher than a preset similarity change threshold, the first radar data and the second radar data corresponding to the visual data corresponding to the structural similarity change data are acquired; wherein, the time of the first radar data is earlier than the time of the second radar data. The first radar point cloud density of the target object is calculated based on the first radar data, and the second radar point cloud density of the target object is calculated based on the second radar data. Calculate the point cloud density change data between the first radar point cloud density and the second radar point cloud density; When the point cloud density of the first radar is higher than that of the second radar, and the point cloud density change data reaches a preset point cloud density threshold, it is determined that the target object is occluded. The occlusion coefficient determination module is also used for: occlusion coefficients are calculated based on the structural similarity change data. Alternatively, the occlusion coefficient can be calculated based on the point cloud density change data; Alternatively, the occlusion coefficient can be calculated based on the structural similarity change data and the point cloud density change data.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-7.
10. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-7.
Citation Information
Patent Citations
Strip mine surface three-dimensional model construction method, device and equipment and storage medium
CN113593017A
Occupied grid map construction method, training method, device and electronic equipment
CN119048704A
