Safety reminding method and device, storage medium and program product

Abstract

The embodiment of the invention provides a safety reminding method and device, a storage medium and a program product, and the method comprises the steps: obtaining first state data of a plurality of key nodes of a user wearing a head-mounted device, obtaining second state data of a safety boundary, and determining a collision prediction result according to the first state data and the second state data, and generating safety reminding information according to the collision prediction result. According to the safety reminding method provided by the embodiment of the invention, collision prediction is performed according to the more detailed data of the poses of the plurality of key nodes of the user and the safety boundary, so that a prediction result with relatively high accuracy can be obtained, the accuracy and effectiveness of safety reminding are improved, the user cannot be disturbed too much, the safety of the user can be effectively guaranteed, and the user experience is improved. And the user experience is improved.

Description

Technical Field

[0001] This disclosure relates to the field of extended reality technology, and in particular to a safety alert method, device, storage medium, and program product. Background Technology

[0002] Users may collide with objects in the real physical environment while using head-mounted devices due to obstructed vision.

[0003] How to accurately and effectively remind users to ensure their safety while minimizing disruption is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This disclosure provides a safety alert method, device, storage medium, and program product to improve the accuracy and effectiveness of safety alerts and reduce disruption to users using head-mounted devices.

[0005] In a first aspect, embodiments of this disclosure provide a security alert method, including:

[0006] Acquire first state data of multiple key nodes of a user wearing a head-mounted device; the first state data includes the pose of the key nodes;

[0007] Obtain the second state data of the security boundary;

[0008] Based on the first state data and the second state data, the collision prediction result is determined;

[0009] A safety alert is generated based on the collision prediction results.

[0010] Secondly, embodiments of this disclosure provide a security alert device, comprising:

[0011] The acquisition module is used to acquire first state data of multiple key nodes of a user wearing a head-mounted device; the first state data includes the pose of the key nodes;

[0012] The acquisition module is also used to acquire second state data of objects in the real environment; the second state data includes the position and surface information of the objects;

[0013] The determination module is used to determine the collision prediction result based on the first state data and the second state data;

[0014] The generation module is used to generate safety reminder information based on the collision prediction results.

[0015] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0016] The memory stores computer-executed instructions;

[0017] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the security alert method as described in the first aspect and various possible designs of the first aspect.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the security alert method described in the first aspect and various possible designs of the first aspect.

[0019] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the security alert method described in the first aspect and various possible designs of the first aspect.

[0020] This embodiment provides a safety alert method, device, storage medium, and program product. The method includes acquiring first state data of multiple key nodes of a user wearing a head-mounted device, the first state data including the pose of the key nodes; acquiring second state data of a safety boundary; determining a collision prediction result based on the first and second state data; and generating a safety alert message based on the collision prediction result. The safety alert method provided in this embodiment, by using more detailed data on the pose of multiple key nodes of the user and comparing it with the safety boundary, can obtain highly accurate prediction results, improving the accuracy and effectiveness of safety alerts. It achieves the goal of not unduly disturbing the user while effectively protecting user safety and enhancing the user experience. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of the security alert method provided in the embodiments of this disclosure;

[0023] Figure 2 Flowchart of the safety reminder method provided in this embodiment of the disclosure Figure 1 ;

[0024] Figure 3 Structural schematic diagrams of several key points provided in the embodiments of this disclosure;

[0025] Figure 4Flowchart of the security alert method provided in this embodiment of the disclosure Figure 2 ;

[0026] Figure 5 A schematic diagram illustrating the principle of the method for determining fault prediction results provided in this embodiment of the disclosure;

[0027] Figure 6 This is a structural block diagram of a safety alert device provided in an embodiment of the present disclosure.

[0028] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] When users use extended reality (XR) headsets such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), they can easily become immersed in the virtual reality and lose their awareness of the surrounding real physical environment, which may lead to accidental collisions with objects in the displayed world while moving.

[0031] In related technologies, safety boundary technology is typically employed to ensure user safety when using head-mounted devices. Specifically, users can move freely within the safety boundary. Based on the head-mounted device's pose, the distance between it and the pre-set safety boundary is determined. When the user approaches or leaves the safety boundary, the head-mounted device's perspective effect is activated, allowing the user to see the real external environment and thus avoiding collisions. However, this method often displays the real environment even when the user will not collide with obstacles, resulting in lower accuracy and effectiveness of safety alerts and higher user intrusion.

[0032] To address the aforementioned technical problems, the inventors of this disclosure have discovered that, compared to providing alerts based on the pose and safety boundaries of a head-mounted device to reveal the real-world environment, a more accurate method can be developed based on the user's pose at key points to predict the area where a collision is imminent. This allows for targeted exposure of relevant areas, providing more effective alerts to the user while minimizing disruption and ensuring user safety. Based on this, embodiments of this disclosure provide a safety alert method.

[0033] Figure 1 This is a schematic diagram illustrating an application scenario of the security alert method provided in this embodiment of the disclosure. For example... Figure 1 As shown, a user (not showing an overall human body diagram, only key skeletal nodes) is wearing a head-mounted device 101. Obstacles (stationary or moving objects) exist in the space where the user is located. The head-mounted device 101 can be an all-in-one device and can be connected to terminal devices such as computers; this embodiment does not limit this.

[0034] In the specific implementation process, taking the head-mounted device 101 as an all-in-one device as an example, the head-mounted device 101 acquires first state data of multiple key nodes of the user wearing the head-mounted device. The first state data includes the pose of the key nodes. It also acquires second state data of the safety boundary. Based on the first and second state data, it determines the collision prediction result and generates a safety reminder message based on the collision prediction result. The safety reminder method provided in this embodiment of the disclosure, by using more detailed data such as the pose of multiple key nodes of the user and the safety boundary to perform collision prediction, can obtain a more accurate prediction result, improve the accuracy and effectiveness of the safety reminder, and achieve both minimal disturbance to the user and effective protection of user safety, thereby enhancing the user experience.

[0035] It should be noted that, Figure 1 The schematic diagram shown is merely an example. The security reminder method and scenario described in this application embodiment are intended to more clearly illustrate the technical solution of this application embodiment and do not constitute a limitation on the technical solution provided in this application embodiment. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solution provided in this application embodiment is also applicable to similar technical problems.

[0036] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0037] Figure 2 Flowchart of the safety reminder method provided in this embodiment of the disclosure Figure 1 The method in this embodiment can be applied to terminal devices (tablet computers, computers, or similar devices). Figure 1In the head-mounted device (or similar device) or server shown, the security alert method includes:

[0038] 201. Obtain first state data of multiple key nodes of a user wearing a head-mounted device; the first state data includes the pose of the key nodes.

[0039] In this embodiment of the disclosure, multiple key nodes may include human skeletal nodes, such as... Figure 1 As shown, nodes may include the torso (which may include the chest, buttocks, etc.) represented by black dots, the upper limbs (which may include the elbows, hands, etc.) represented by green dots, the lower limbs (which may include the feet, knees, etc.) represented by blue dots, and the head represented by red dots.

[0040] In one embodiment of this disclosure, considering that the pose of the key node is only static data such as position and orientation, dynamic data can be added to improve the accuracy of collision prediction. Specifically, the first state data may also include the motion information of the key node. The motion information of the key node can be determined by recognizing multiple consecutive images, or by determining the motion information of the head-mounted device (or the target device such as a handheld device) based on sensor data collected by sensors such as accelerometers and inertial measurement units installed on the head-mounted device (or the target device such as a handheld device). The motion information of the key node is then determined based on this motion information.

[0041] In this embodiment of the disclosure, there are multiple ways to determine the first state data of multiple key nodes.

[0042] In one possible implementation, acquiring the first state data of multiple key nodes of a user wearing a head-mounted device may include: acquiring a target image captured by the head-mounted device; the target image including multiple key nodes; and parsing the target image based on a deep learning model to obtain the first state data of the multiple key nodes.

[0043] Specifically, a downward-facing camera (meaning the camera on the head-mounted device is oriented downwards when the user is standing and looking straight ahead) can be installed on the head-mounted device. When the user is active while wearing the device, this camera captures images that show the human body and / or accessories held or worn by the user (e.g., rackets, baseball bats). By analyzing these images, the poses and motion information (velocity, acceleration, etc.) of multiple key nodes of the human body or accessories can be determined. This allows for the estimation of the trajectories of each key node for collision prediction. For example, the images can be input into a deep learning model for image analysis. This deep learning model is trained using multiple historical images, which include different poses of multiple key nodes.

[0044] In another possible implementation, acquiring the first state data of multiple key nodes of a user wearing a head-mounted device may include: acquiring third state data of the head-mounted device and fourth state data of a target device associated with the head-mounted device; the third state data includes the pose of the head-mounted device; and determining the first state data of multiple key nodes of the interactor based on the third state data and the fourth state data using an inverse kinematics algorithm.

[0045] The third state data can include the pose and motion information of the head-mounted device. The association between the head-mounted device and the target device means that the two can be used together (e.g., worn simultaneously on the user, changing pose with the user's movement), or that there is data interaction between them.

[0046] In this embodiment, the target device includes at least one of the following: a first device for mounting on the user's upper limb, a second device for mounting on the user's lower limb, and a third device for mounting on a user-carrying accessory. In one implementation, the target device (e.g., a handle) may be mounted on the user's upper limb, and the fourth state data includes the pose and motion information of the target device. In another implementation, the target device includes a first device (e.g., a handle, a motion capture device with an independent tracking mode) and a second device (e.g., a motion capture device with a full-body motion capture mode, an IMU sensor), with the first device mounted on the user's upper limb and the second device mounted on the user's lower limb; the fourth state data includes the pose of the first device and the posture of the second device. In yet another implementation, the target device, in addition to the above two implementations, further includes a third device (e.g., a motion capture device) mounted on a user-carrying accessory (e.g., a racket or baseball bat).

[0047] In another possible implementation, acquiring first state data of multiple key nodes of a user wearing a head-mounted device may include: acquiring first sensing data of the head-mounted device and second sensing data of a target device associated with the head-mounted device; the target device includes a first device for placement on the upper limb of the user's body, and / or a second device for placement on the lower limb of the user's body; and determining the first state data of multiple key nodes of the interactor based on a multimodal algorithm, according to the first sensing data and the second sensing data.

[0048] Specifically, the head-mounted device can be equipped with visual sensors, time-of-flight sensors, inertial measurement units, gyroscopes, magnetometers, accelerometers, and Global Positioning System (GPS) sensors. The target device can be equipped with accelerometers, gyroscopes, magnetometers, optical sensors, touch sensors, eye-tracking sensors, and audio sensors. Through multimodal algorithms, the pose, location, and motion information of multiple key nodes can be obtained directly using the sensor data collected by the sensors of both the head-mounted device and the target device. This implementation method enables the direct acquisition of the pose of multiple key nodes from sensor data, improving computational efficiency and accuracy.

[0049] In one embodiment of this disclosure, the plurality of key nodes may include skeletal nodes of the user's body, and / or key nodes of accessories carried by the user.

[0050] like Figure 3 As shown, the key nodes include not only the skeletal nodes of the human body (represented by black dots) but also the key nodes of the racket (represented by yellow dots). In one implementation, when a user holds the racket to play, a target device (e.g., a motion capture device) can be set on the racket. Since the target device and the racket have a fixed positional relationship, the pose of the racket's key nodes can be obtained based on the target device, and the shape of the racket can be matched. In another implementation, images can be acquired using a downward-facing camera on a head-mounted device. The images can include the racket's key nodes, and the pose of the racket's key nodes can be determined based on the acquired images.

[0051] 202. Obtain the second state data of the security boundary.

[0052] Specifically, security boundaries can be pre-set virtual boundaries or real boundaries formed by the surfaces of real objects in a real environment.

[0053] In the first embodiment of this disclosure, the security boundary is a pre-set virtual boundary; the second state data includes the boundary information of the virtual boundary.

[0054] Specifically, the virtual boundary can be user-defined or set by the head-mounted device's system based on a preset algorithm. This virtual boundary can be a cylindrical boundary perpendicular to the plane of the ground. The ground surface of the cylindrical boundary can be a regular shape, such as a cylinder, or an irregular shape, such as a cylinder with an irregular base generated based on user drawing or ground obstacles.

[0055] In one embodiment of this disclosure, the safety boundary is a real boundary formed by an object in a real environment; the second state data includes surface information of the object in the real environment. Specifically, obtaining the second state data of the safety boundary includes: obtaining the second state data of the object in the real environment; the second state data includes surface information of the object.

[0056] Specifically, objects in the real environment can be living beings such as people and pets, or inanimate objects such as tables, chairs, sofas, walls, and strollers.

[0057] In one embodiment of this disclosure, the surface of objects in the real environment can be detected using all sensors on the head-mounted device. This detection can be formula-based, machine learning-based, user-calibrated, user-informed, automatically detected when danger occurs, or even globally networked, based on information or calibration from other users. In the future, outdoor applications can also be calculated based on internet maps or panoramic images, and the position of cars can be calculated based on real-time traffic flow. Obstacles, buildings, and cars in the outdoors can be detected in advance. More reasonable interaction collision mechanisms for outdoor environments can also be designed, such as early detection and alerts for approaching cars.

[0058] In one embodiment of this disclosure, obtaining second state data of an object in a real environment may include: acquiring third sensor data (visual sensor, time-of-flight sensor, etc.) from a head-mounted device; and performing mesh reconstruction on the surface of the object in the real environment based on the position information of the object's surface to obtain the second state data of the object in the real environment. Optionally, performing mesh reconstruction on the surface of the object in the real environment based on the position information of the object's surface to obtain the second state data of the object in the real environment may include: determining the position information of the object's surface in the real environment based on the third sensor data using a neural network model; and reconstructing the surface of the object in the real environment based on the position information of the object's surface using spatial mesh technology to obtain the second state data of the object in the real environment.

[0059] Specifically, third-sensor data can be acquired using all the sensors in the head-mounted device, and a mesh reconstruction of the surface of objects in the real environment can be performed based on this data. To improve efficiency, computer vision technology based on neural network learning can be combined during the mesh reconstruction process to detect the positions of all object surfaces in the real physical space. For objects with only partial surfaces, partial surface reconstruction technology (the process of recovering the object's surface from a series of observed images or other types of data) can be used to recover the object's surface, and a mesh can be reconstructed for the object's surface in space based on spatial mesh technology. The mesh, as a series of triangular facet information, can be used to calculate the location and surface information of the object in space.

[0060] 203. Determine the collision prediction result based on the first state data and the second state data.

[0061] Specifically, after acquiring the poses of multiple key nodes and the positions and surface information of objects in real physical space, the positional relationships between these key nodes and the surfaces of neighboring objects can be monitored. Based on these positional relationships, the likelihood of a collision, as well as the timing and location of the collision, can be predicted, yielding a collision prediction result. This result can include the predicted location of the collision and the time of its occurrence. Since this collision prediction result is determined based on the poses of the key nodes, it is equivalent to determining the collision prediction result based on a more detailed part of the human body (such as the foot, knee, or elbow). Compared to treating the head-mounted device or controller as a whole, determining the collision prediction result based on the pose of the head-mounted device or controller results in a more accurate outcome.

[0062] 204. Generate a safety reminder message based on the collision prediction results.

[0063] Specifically, after obtaining the collision prediction results, safety alerts can be generated based on the predicted location (and time point) indicated by the collision prediction results. The main purpose of safety alerts is to remind users of impending dangers. Safety alerts can take many forms, such as displaying virtual grids in space, displaying safety boundary fences to alert users, revealing the real environment to show users the actual dangers, or displaying prompts. For greater visibility, safety alerts can be placed at the predicted collision location, such as displaying grid data at the predicted location, displaying the safety boundary fence at the predicted location, revealing the real environment at the predicted location, or displaying prompts at the predicted location. It should be noted that, to avoid interrupting user use, the head-mounted device also displays virtual content. Safety alerts such as virtual grids, safety boundary fences, and prompts can be displayed together with this virtual content. When revealing the environment to provide safety alerts, it is also done partially from the virtual content to minimize disruption to user experience.

[0064] In one embodiment of this disclosure, different safety reminder messages can be generated based on different indoor and outdoor scenarios, or the user can set them themselves. The dangerous scene or object can be displayed transparently or semi-transparently, or other colors, animation effects, grids, etc., can be used to represent the dangerous scene, or there can be more obvious pop-up prompts to the user, or other voice prompts, etc.

[0065] In one embodiment of this disclosure, the collision prediction result includes the predicted location of the collision, and the generation of safety alert information based on the collision prediction result may include at least one of the following: First, displaying grid data of the corresponding location on the surface of an object in the real environment based on the predicted location. Second, revealing the real environment at the corresponding location based on the predicted location. Third, displaying a prompt message at the predicted location.

[0066] As can be seen from the above description, the safety reminder method provided in this disclosure can obtain highly accurate prediction results by using more detailed data such as the poses of multiple key nodes of the user and the safety boundary to perform collision prediction, thereby improving the accuracy and effectiveness of safety reminders. This achieves the goal of not disturbing the user too much while effectively protecting the user's safety and enhancing the user experience.

[0067] Figure 4 Flowchart of the security alert method provided in this embodiment of the disclosure Figure 2 Based on the above-disclosed embodiments, for example in Figure 2 Based on the disclosed embodiments, this disclosure provides a detailed description of the method for generating collision prediction results. The method includes:

[0068] 401. Obtain first state data of multiple key nodes of a user wearing a head-mounted device; the first state data includes the pose of the key nodes.

[0069] 402. Obtain the second state data of the security boundary.

[0070] In this embodiment of the disclosure, steps 401 to 402 are similar to steps 201 to 202 in the above-disclosed embodiments, and will not be described again here.

[0071] 403. Determine the predicted feature value of the collision based on the first state data and the second state data.

[0072] 404. Determine whether the predicted feature value meets the preset conditions. If yes, proceed to step 405; otherwise, return to step 401.

[0073] 405. Generate collision prediction results based on the predicted feature values.

[0074] Specifically, after obtaining the initial state data of multiple key nodes and the position and surface information of objects in the real physical space, it is possible to calculate whether a key node is about to collide with the surface of an object in the real physical space. To improve flexibility, a filtering mechanism can be introduced to determine the degree of impending collision, i.e., at what level a collision prediction result is generated, to implement different calculation methods. The design of the filtering mechanism can take into account various factors, such as human body status, processing latency (the time consumed by processing before the safety alert information is pushed to the user), the user's current intention in using the head-mounted device, and time and distance dimensions.

[0075] In one embodiment of this disclosure, determining a collision prediction result based on the first state data and the second state data may include: determining a predicted feature value for a collision to occur based on the first state data and the second state data; and generating a collision prediction result based on the predicted feature value if the predicted feature value satisfies a preset condition.

[0076] Specifically, a predictive feature value can be determined first, such as the time or location of the collision. Then, filtering conditions can be set for this predictive feature value. When the preset conditions are met, the collision prediction result is generated.

[0077] In one possible implementation, determining the predicted feature value of the collision based on the first state data and the second state data may include: determining the predicted location of the collision and the corresponding key node of the predicted collision based on the first state data and the second state data; and generating a collision prediction result based on the predicted feature value if the predicted feature value meets a preset condition may include: generating a collision prediction result based on the predicted location if the distance between the predicted location and the key node of the predicted collision is less than a preset distance.

[0078] For example, such as Figure 5 As shown, after obtaining the first state data (pose and / or motion information) of the key foot nodes, the motion trajectory can be predicted based on this first state data, and then the predicted collision location can be determined based on the motion trajectory. The distance between the predicted location and the predicted key node of the collision can be a straight-line distance or the length of the trajectory along the predicted path. If the distance between the predicted location and the predicted key node of the collision is less than a preset distance, it indicates that the risk factor is already high, and the user needs to be alerted. Therefore, a collision prediction result is generated.

[0079] In another possible implementation, determining the predicted feature value of the collision based on the first state data and the second state data may include: determining the predicted time point and predicted location of the collision based on the first state data and the second state data; if the duration between the current time point and the predicted time point is less than a preset duration, then generating a collision prediction result based on the predicted time point and predicted location.

[0080] For example, such as Figure 5 As shown, after obtaining the first state data (pose and / or motion information) of key footstep nodes, the motion trajectory can be predicted based on this first state data. Then, based on the motion trajectory, the predicted collision time point is determined. If the time interval between the current time point and the predicted time point is less than a preset time interval, it indicates that the risk factor is already high, and the user needs to be alerted. Therefore, a collision prediction result is generated.

[0081] In one embodiment of this disclosure, considering that the human body's reaction sensitivity varies depending on its state, the timeliness of collision alerts can differ to improve user experience. Specifically, if the predicted feature value meets a preset condition, before generating a collision prediction result based on the predicted feature value, the process may further include: acquiring the user's state information; the state information including the user's action sensitivity; and determining the preset condition based on the state information.

[0082] Specifically, for predicting a person's state: machine learning solutions can be introduced, or the user can choose their own approach. For example, user profiles can be determined based on usage data from the head-mounted device. This includes identifying the time periods when a user prefers activities requiring high sensitivity (such as shooting games or boxing), and the effectiveness of those activities; and the time periods when they prefer activities requiring lower sensitivity (such as watching movies or walking through 3D environments), and the effectiveness of those activities. Based on these user profiles, factors associated with the user's state information (time of day, type of game played, etc.) can be identified. Different combinations of factors correspond to different states. When a certain combination of factors occurs, the user is determined to be in the corresponding state, thus establishing the corresponding preset conditions. Generally, the better the state, the more difficult it is to meet the preset conditions because the user's reaction speed is fast, and no special advance warning is needed.

[0083] In one embodiment of this disclosure, determining a collision prediction result based on the first state data and the second state data may include: acquiring user interaction data in a head-mounted device; determining the user's usage intention based on the interaction data; and if the user's usage intention is a preset intention, adjusting the preset conditions to increase the generation frequency of collision prediction results.

[0084] Interaction data can include virtual consumption information (purchases of boxing gloves, movie tickets, etc.) and the type of application being used (e.g., boxing games requiring many movements versus movie-watching applications requiring less movement). Based on this interaction data, user intent can be determined. If a user's intent is to play a boxing match, the risk of collisions increases, allowing for early warnings. Conversely, if the user's intent is to watch a movie, collisions are unlikely, so warnings can be reduced or collision detection can be disabled to conserve resources, reactivating the collision detection function when the user's intent changes.

[0085] 406. Generate a safety reminder message based on the collision prediction results.

[0086] In this embodiment of the disclosure, step 406 is similar to step 204 in the above-disclosed embodiment, and will not be described again here.

[0087] As can be seen from the above description, the safety reminder method provided in this disclosure embodiment uses a screening mechanism to filter and generate collision prediction results based on predicted feature values, so as to achieve a balance between reducing disturbance and improving safety.

[0088] Corresponding to the security reminder method in the above embodiments, Figure 6 This is a structural block diagram of a safety alert device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 6The device includes: an acquisition module 601, a determination module 602, and a generation module 603.

[0089] The acquisition module 601 is used to acquire first state data of multiple key nodes of a user wearing a head-mounted device; the first state data includes the pose of the key nodes.

[0090] The acquisition module 601 is also used to acquire the second state data of the security boundary;

[0091] The determining module 602 is used to determine the collision prediction result based on the first state data and the second state data;

[0092] The generation module 603 is used to generate safety reminder information based on the collision prediction results.

[0093] In one embodiment of this disclosure, the first state data further includes motion information of key nodes.

[0094] In one embodiment of this disclosure, when the acquisition module 601 acquires first state data of multiple key nodes of a user wearing a head-mounted device, it is specifically used to: acquire a target image collected by the head-mounted device; the target image includes multiple key nodes; and parse the target image based on a deep learning model to obtain the first state data of the multiple key nodes.

[0095] In one embodiment of this disclosure, when the acquisition module 601 acquires first state data of multiple key nodes of a user wearing a head-mounted device, it is specifically used to: acquire third state data of the head-mounted device and fourth state data of a target device associated with the head-mounted device; the third state data includes the pose of the head-mounted device; and determine the first state data of multiple key nodes of the interactor based on the inverse kinematics algorithm, according to the third state data and the fourth state data.

[0096] In one embodiment of this disclosure, the target device includes at least one of the following: a first device for placement on a user's upper limb, a second device for placement on a user's lower limb, and a third device for placement on a user-carrying accessory.

[0097] In one embodiment of this disclosure, when the acquisition module 601 acquires first state data of multiple key nodes of a user wearing a head-mounted device, it is specifically used to: acquire first sensing data of the head-mounted device and second sensing data of a target device associated with the head-mounted device; the target device includes a first device for installation on the upper limb of the user's body, and / or a second device for installation on the lower limb of the user's body; and, based on a multimodal algorithm, determine the first state data of multiple key nodes of the interactor according to the first sensing data and the second sensing data.

[0098] In one embodiment of this disclosure, the plurality of key nodes include skeletal nodes of the user's body, and / or key nodes of accessories carried by the user.

[0099] In one embodiment of this disclosure, the security boundary is a pre-set virtual boundary; the second state data includes the boundary information of the virtual boundary.

[0100] In one embodiment of this disclosure, the security boundary is a real boundary formed by objects in a real environment; the second state data includes surface information of the objects in the real environment.

[0101] In one embodiment of this disclosure, when acquiring the second state data of the safety boundary, the acquisition module 601 is specifically used to: acquire the third sensing data of the head-mounted device; and reconstruct the surface of the object in the real environment based on the position information of the surface of the object in the real environment to obtain the second state data of the object in the real environment.

[0102] In one embodiment of this disclosure, when determining the collision prediction result based on the first state data and the second state data, the determining module 602 is specifically used to: determine the predicted feature value of the collision occurrence based on the first state data and the second state data; if the predicted feature value meets a preset condition, then generate the collision prediction result based on the predicted feature value.

[0103] In one embodiment of this disclosure, when determining the predicted feature value of a collision based on the first state data and the second state data, the determining module 602 is specifically used to: determine the predicted location of the collision and the corresponding key node of the predicted collision based on the first state data and the second state data; if the distance between the predicted location and the key node of the predicted collision is less than a preset distance, then generate a collision prediction result based on the predicted location.

[0104] In one embodiment of this disclosure, when determining the predicted feature value of a collision based on the first state data and the second state data, the determining module 602 is specifically used to: determine the predicted time point and predicted location of the collision based on the first state data and the second state data; if the duration between the current time point and the predicted time point is less than a preset duration, then generate a collision prediction result based on the predicted time point and the predicted location.

[0105] In one embodiment of this disclosure, before generating a collision prediction result based on the predicted feature value if the predicted feature value meets the preset conditions, the determining module 602 is further configured to: obtain user state information; the state information includes the user's action sensitivity; and determine the preset conditions based on the state information.

[0106] In one embodiment of this disclosure, when determining the collision prediction result based on the first state data and the second state data, the determining module 602 is specifically configured to: acquire user interaction data in the head-mounted device; determine the user's usage intention based on the interaction data; and if the user's usage intention is a preset intention, adjust the preset conditions to increase the generation frequency of the collision prediction result.

[0107] In one embodiment of this disclosure, when the collision prediction result includes the predicted location of the collision, the generation module 603, when generating safety reminder information based on the collision prediction result, specifically performs at least one of the following: based on the predicted location, displaying the grid data of the corresponding location on the surface of the object in the real environment; based on the predicted location, revealing the real environment at the corresponding location; and displaying a prompt message at the predicted location.

[0108] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0109] To implement the above embodiments, this disclosure also provides an electronic device.

[0110] refer to Figure 7 The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0111] like Figure 7As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0112] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0113] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0114] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0115] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0116] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0117] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0120] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0121] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0123] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0124] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A safety reminding method, characterized by, The method comprises: obtaining first state data of a plurality of key nodes of a user wearing a head-mounted device; the first state data comprises a pose of the key nodes; obtaining second state data of a safety boundary; determining a collision prediction result according to the first state data and the second state data; generating safety reminder information according to the collision prediction result.

2. The method of claim 1, wherein, The first state data further comprises motion information of the key nodes.

3. The method of claim 1, wherein, The method of obtaining first state data of a plurality of key nodes of a user wearing a head-mounted device comprises: obtaining a target image collected by the head-mounted device; the target image comprises a plurality of key nodes; based on a deep learning model, the target image is analyzed to obtain the first state data of the plurality of key nodes.

4. The method of claim 1, wherein, The method of obtaining first state data of a plurality of key nodes of a user wearing a head-mounted device comprises: obtaining third state data of the head-mounted device and fourth state data of a target device associated with the head-mounted device; the third state data comprises a pose of the head-mounted device; based on an inverse kinematics algorithm, the first state data of a plurality of key nodes of an interactor is determined according to the third state data and the fourth state data.

5. The method of claim 4, wherein, The target device comprises at least one of the following: a first device for being arranged on an upper limb of the user, a second device for being arranged on a lower limb of the user, and a third device for being arranged on a carrying accessory of the user.

6. The method of claim 1, wherein, The method of obtaining first state data of a plurality of key nodes of a user wearing a head-mounted device comprises: obtaining first sensing data of the head-mounted device and second sensing data of a target device associated with the head-mounted device; the target device comprises a first device for being arranged on an upper limb of the user, and / or a second device for being arranged on a lower limb of the user; based on a multi-modal algorithm, the first state data of a plurality of key nodes of an interactor is determined according to the first sensing data and the second sensing data.

7. The method of claim 1, wherein, The plurality of key nodes comprises a skeletal node of the user's body and / or a key node of a carrying accessory of the user.

8. The method according to any one of claims 1 to 7, characterized in that, The safety boundary is a pre-set virtual boundary; the second state data comprises boundary information of the virtual boundary.

9. The method according to any one of claims 1 to 7, characterized in that, The safety boundary is a real boundary formed by an object in a real environment; the second state data comprises surface information of the object in the real environment.

10. The method of claim 9, wherein, The method of obtaining second state data of a safety boundary comprises: obtaining third sensing data of the head-mounted device; reconstructing the surface of the object in the real environment according to position information of the surface of the object in the real environment to obtain second state data of the object in the real environment.

11. The method according to any one of claims 1 to 7, characterized in that, The method of determining a collision prediction result according to the first state data and the second state data comprises: determining a predicted characteristic value of collision occurrence according to the first state data and the second state data; if the predicted characteristic value meets a preset condition, a collision prediction result is generated according to the predicted characteristic value.

12. The method of claim 11, wherein, The method of determining a predicted characteristic value of collision occurrence according to the first state data and the second state data comprises: determining a predicted position of collision occurrence and a corresponding predicted key node of collision occurrence according to the first state data and the second state data; If the predicted feature value meets a preset condition, then generating a collision prediction result based on the predicted feature value includes: If the distance between the predicted location and the predicted key node where a collision is to occur is less than a preset distance, then a collision prediction result is generated based on the predicted location.

13. The method of claim 11, wherein, The step of determining the predicted feature value of the collision occurrence based on the first state data and the second state data includes: Based on the first state data and the second state data, the predicted time point and predicted location of the collision are determined; If the predicted feature value meets a preset condition, then generating a collision prediction result based on the predicted feature value includes: If the time interval between the current time point and the predicted time point is less than the preset time interval, a collision prediction result is generated based on the predicted time point and the predicted location.

14. The method of claim 11, wherein, Before generating a collision prediction result based on the predicted feature value if the predicted feature value meets a preset condition, the method further includes: Obtain the user's status information; the status information includes the user's action sensitivity. The preset conditions are determined based on the status information.

15. The method of claim 11, wherein, Determining the collision prediction result based on the first state data and the second state data includes: Acquire user interaction data within the head-mounted device; Determine the user's intent based on the interaction data; If the user's intent is a preset intent, the preset conditions are adjusted to increase the frequency of collision prediction results.

16. The method according to any one of claims 1 to 7, characterized in that, The collision prediction result includes the predicted location of the collision, and the generation of safety alert information based on the collision prediction result includes at least one of the following: Based on the predicted location, display the grid data of the corresponding location on the surface of the object in the real environment; Based on the predicted location, the actual environment at the corresponding location is revealed; A prompt message is displayed at the predicted location.

17. A safety reminder device, characterized by include: The acquisition module is used to acquire the first state data of multiple key nodes of the user wearing the head-mounted device; The first state data includes the poses of key nodes; The acquisition module is also used to acquire the second state data of the security boundary; The determination module is used to determine the collision prediction result based on the first state data and the second state data; The generation module is used to generate safety reminder information based on the collision prediction results.

18. An electronic device, comprising: include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the security alert method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the security alert method as described in any one of claims 1 to 16.

20. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the security alert method as described in any one of claims 1 to 16.

Images