Information prompting method and apparatus, and extended reality device, medium and program product

Abstract

An information prompting method and apparatus, and an extended reality device, a medium and a program product. The method comprises: when a first subject wears an extended reality device, displaying a virtual scene picture, wherein a first active region in a physical scene is a safe active region of the first subject; and when a second subject is located in the first active region, in response to the distance between a first location of the first subject and a second location of the second subject meeting a prompting requirement, displaying prompt information or a second scene picture, wherein the prompt information or the second scene picture is used for providing to the first subject a prompt of a collision relationship with the second subject.

Description

Information prompt method, extended reality device, apparatus, medium and program product

[0001] This application claims priority to Chinese patent application number 202410389974.7, filed on April 1, 2024, entitled “Information prompt method, extended reality device, apparatus, medium and program product,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of the present application relate to the field of virtual reality, and in particular to an information prompting method, an extended reality device, an apparatus, a medium, and a program product. Background Art

[0003] Extended display devices are used to create and simulate a three-dimensional virtual world. Users can immerse themselves in VR games by wearing extended reality devices. While wearing extended reality devices, users cannot see their surroundings. During the game experience, users may touch objects in the surrounding environment when moving or performing different actions.

[0004] In related technologies, extended reality devices will prompt users to pre-set a safe area and constrain the user's movement range based on the safe area. When the user moves to the boundary of the safe area or exceeds the range of the safe area, the user will be immediately prompted to return in time.

[0005] However, the above method can only ensure the safety of users when there are only static objects in the surrounding environment, which has limitations. When there are other people passing by in the surrounding environment, players may have physical collisions with other people during the game and be injured. Summary of the Invention

[0006] The embodiments of the present application provide an information prompt method, an extended reality device, an apparatus, a medium, and a program product that can predict the motion trajectories of a first subject and a second subject, provide information prompts to the first subject wearing the extended reality device, avoid collisions between the first subject and the second subject, and ensure the safety of the first subject. The technical solution is as follows:

[0007] In one aspect, a method for providing information prompts based on an extended reality device is provided, the method comprising:

[0008] When a first subject wears the extended reality device, a virtual scene image is displayed, and a first activity area in the physical scene is a safe activity area for the first subject;

[0009] When the second subject is located within the first activity area, in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirements, a prompt message or a second scene picture is displayed, and the prompt message or the second scene picture is used to prompt the first subject of the collision relationship with the second subject.

[0010] In another aspect, an extended reality device is provided, the extended reality device comprising a display screen assembly and a control chip;

[0011] The control chip is used to control the display screen assembly to display a virtual scene image when a first subject wears the augmented reality device, wherein the virtual scene image is used to provide VR scene perception to the first subject, and the first subject corresponds to the first activity area;

[0012] The control chip is also used to control the display screen assembly to display prompt information or a second scene picture in response to the distance between the first position of the first body and the second position of the second body meeting the prompt requirement when the second body is located in the first active area. The prompt information or the second scene picture is used to prompt the first body of the collision relationship with the second body.

[0013] In another aspect, an information prompting device based on an extended reality device is provided, the device comprising:

[0014] A scene screen display module, configured to display a virtual scene screen when a first subject wears the augmented reality device, wherein the first activity area in the physical scene is a safe activity area for the first subject;

[0015] A prompt information display module is configured to display prompt information or a second scene image when a second subject is located within the first active area and in response to the distance between the first position of the first subject and the second position of the second subject meeting a prompt requirement, wherein the prompt information or the second scene image is configured to notify the first subject of a collision relationship with the second subject. In another aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the information prompt method as described in any of the above-mentioned embodiments of the present application.

[0016] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the information prompt method described in any of the above embodiments.

[0017] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0018] When a first subject wears an augmented reality device, a virtual scene is displayed, and a first activity area in the physical scene is the first subject's safe activity area. When a second subject is within the first activity area, a prompt message or a second scene is displayed in response to the distance between the first subject's first position and the second subject's second position meeting the prompt requirement. The prompt message or the second scene is used to inform the first subject of a collision relationship with the second subject. The distance between the first subject and the second subject can be used to predict a possible collision between the first subject and the second subject, and based on the prompt message, the first subject is promptly instructed to adjust its movement trajectory to avoid a collision and ensure the safety of the first subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] FIG1 is an overall flow chart of the present solution provided by an exemplary embodiment of the present application;

[0021] FIG2 is a schematic diagram of an extended reality device provided by an exemplary embodiment of the present application;

[0022] FIG3 is a flowchart of an information prompting method based on an extended reality device provided by an exemplary embodiment of the present application;

[0023] FIG4 is a schematic diagram of a stitched image provided by an exemplary embodiment of the present application;

[0024] FIG5 is a flow chart of a method for determining the presence of a second subject in a first activity area provided by an exemplary embodiment of the present application;

[0025] FIG6 is a schematic diagram showing a first relative positional relationship between a second body and a first body provided by an exemplary embodiment of the present application;

[0026] FIG7 is a schematic diagram of determining a first relative position relationship based on an extension line provided by an exemplary embodiment of the present application;

[0027] FIG8 is a flow chart of a method for collecting first motion trajectory data of a first subject and second motion trajectory data of a second subject provided by an exemplary embodiment of the present application;

[0028] FIG9 is a schematic diagram of mapping the position of the center of the second subject in multiple frames of environmental images to a three-dimensional coordinate system provided by an exemplary embodiment of the present application;

[0029] FIG10 is a schematic diagram showing the size relationship between the first active area and the second active area provided by an exemplary embodiment of the present application;

[0030] FIG11 is a schematic diagram of second predicted trajectory data provided by an exemplary embodiment of the present application;

[0031] FIG12 is a structural block diagram of an information prompting apparatus based on an extended reality device provided by an exemplary embodiment of the present application;

[0032] FIG13 is a structural block diagram of an information prompting device based on an extended reality device provided by another exemplary embodiment of the present application;

[0033] FIG14 is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0035] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0036] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0037] It should be noted that the information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0038] It should be understood that although the terms first, second, etc. may be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0039] A brief introduction to the terms involved in the embodiments of this application is given below:

[0040] Virtual Reality (VR) is a computer simulation technology that enables the creation and experience of virtual worlds. It uses computers to generate interactive, three-dimensional, dynamic scenes, immersing users in and allowing them to interact with them through specialized equipment. These devices, including head-mounted displays, isolate the user's visual and auditory senses from the outside world, allowing them to experience a sense of presence within the virtual environment.

[0041] The extended reality device referred to in this application refers to a VR device, and the extended display device is a head-mounted display device. When a user wears the extended reality device, the image they see changes from the real environment image to the virtual environment image generated by the extended reality device.

[0042] Field of view: The angle formed by the two edges of the maximum range through which the image of the object being measured can pass through the lens of an optical instrument, with the lens as the vertex. This is also the angle between the edge of the display and the observation point (eye). In the VR field, field of view usually refers to the horizontal field of view.

[0043] RGB (Red, Green, Blue color model): In terms of image acquisition, RGB cameras can capture the color information of the scene, while grayscale cameras can only capture brightness information. Each pixel in an RGB image is represented by the values ​​of three color components (R, G, B), while each pixel in a grayscale image has only one brightness value, which is usually obtained by converting the RGB components into a single grayscale value. Among them, the grayscale camera component involved in this application is a grayscale camera, and the RGB depth camera component is an RGB depth camera.

[0044] Extended reality devices are widely used in the gaming field. After wearing extended reality devices, users can experience a variety of VR games, immerse themselves in a virtual three-dimensional environment, interact with game scenes / game characters / game elements, etc., and gain an immersive experience.

[0045] Common VR game types include stationary games and motion games. Stationary games don't require users to move their bodies, requiring only small movements, such as hand movements or body swaying. Motion games, on the other hand, allow users to move freely within a certain space, changing their position to experience the same movement in the virtual world.

[0046] However, the images seen by users wearing XR devices are virtual images presented by the XR device, and users cannot see their surroundings in real time. If users move around extensively while playing mobile games, they may touch objects, creatures, or other obstacles in the real environment, causing physical injury. If users swing their arms while playing stationary games, they may also touch other creatures (such as animals or other users) passing by, posing a safety hazard to users of XR devices.

[0047] In related technologies, some extended reality devices will prompt users to set a safe area in advance, or set a default safe area for users, and constrain the user's movement range based on the safe area. When the user moves to the boundary of the safe area or exceeds the range of the safe area, the user will be immediately prompted to return in time.

[0048] While these prompts can ensure user safety to a certain extent, they also have limitations: the safety zone is determined only by the XR device, and users are unaware of their own location and what's happening in the real world while playing. If someone else enters the user's safety zone, there's a high risk of physical collision and injury.

[0049] This application provides an information prompt method based on an extended reality device, which can effectively prevent users from colliding with other creatures in the real environment when wearing the extended reality device to play games, thereby ensuring the safety of users.

[0050] The extended reality device is presented as shown in Figure 1. Figure 1 is an overall flow chart of the present solution, which includes the following steps and is executed by the extended reality device. The extended reality device includes a camera component that can capture images of the surrounding environment. The user wearing the extended reality device is the first subject.

[0051] S101: Determine a first active area.

[0052] The first activity area is a safe area for the first subject to carry out activities. When the first subject's position leaves the first activity area, the extended reality device will switch the virtual reality screen to the real environment screen to prompt the first subject to return to the first activity area in time.

[0053] S102, loading the game.

[0054] Optionally, the extended reality device is a VR glasses, and when the game is loaded, the game screen will be presented in front of the first subject.

[0055] Optionally, after the first user wears the extended reality device, the game loading can be triggered in the following ways: (1) using a specific gesture or action to trigger, such as reaching out to trigger; (2) triggering the loading of the game by triggering a target control on the extended reality device, such as long pressing the power button on the extended reality device to trigger; (3) using a controller that is compatible with the VR glasses, which includes multiple controls. S103, determining the game type.

[0056] Among them, the game types include: (1) stationary activity games, and (2) mobile games. When playing stationary activity games, users do not need to move their body position, but only need to make small movements of the limbs, such as hand movements, shaking the body in place, etc. When playing mobile games, users can move freely within a certain space and change their position to get the experience of corresponding movement in the virtual world.

[0057] In some embodiments, as soon as the first subject puts on the extended reality device, the extended reality device automatically presents a game type selection interface to the first subject. The game type selection interface displays a variety of different types of VR games. The first subject can also select the game type through the several methods shown in the above step S102.

[0058] S104: If the game type is an on-site activity game, run the game.

[0059] For example, during the running of the game, the extended reality device will display the game interface to the first subject.

[0060] S105: Determine whether there is a second subject in the first activity area.

[0061] If the second subject does not exist in the first active area, then step S104 to step S105 are continued.

[0062] S106: If there is a second subject in the first activity area, motion trajectory data of the first subject is predicted.

[0063] At this time, the game experienced by the first subject is an in-place activity game, the body position of the first subject does not change, and the motion trajectory data of the first subject is used to describe the motion trajectory of the first subject's hand in the future time period.

[0064] Optionally, a circular area is formed by dividing the first body's arm length into a radius and the first body's position into a center, and the edge trajectory of the circular area is the motion trajectory of the first body.

[0065] S107: If the game type is a mobile game, run the game.

[0066] For example, during the running of the game, the extended reality device will display the game interface to the first subject.

[0067] S108: Determine whether there is a second subject in the first activity area.

[0068] If the second subject does not exist in the first active area, then step S107 to step S108 are continued.

[0069] S109: If there is a second subject in the first activity area, predict the motion trajectory data of the first subject.

[0070] At this time, the game experienced by the first subject is a mobile game, and the body position of the first subject will change. The motion trajectory data of the first subject is used to describe the motion trajectory of the first subject in the future time period, that is, the motion trajectory data of the first subject is used to describe the position change of the first subject.

[0071] S110: If there is a second subject in the first activity area, motion trajectory data of the second subject is predicted.

[0072] S111 , determining whether the motion trajectory of the first subject overlaps with the motion trajectory of the second subject.

[0073] It is determined whether there is an intersection between the motion trajectory of the first subject and the motion trajectory of the second subject. If there is an intersection, there is overlap, and the intersection is the overlap position.

[0074] S112: If there is overlap, determine the time when the first body and the second body reach the overlap position respectively.

[0075] Optionally, the moment when the first body reaches the overlapping position is the first moment, and the moment when the second body reaches the overlapping position is the second moment.

[0076] S113: Determine whether the interval between the first moment and the second moment is less than a threshold value.

[0077] If the duration is less than the threshold, it indicates that a collision will occur between the first subject and the second subject.

[0078] S114: If the duration is less than the threshold, cancel the display of the game screen.

[0079] Exemplarily, the duration threshold is 1.5 seconds, the first moment is 10:00:30, the second moment is 10:00:31, and the interval between the first moment and the second moment is 1 second, which is less than the duration threshold.

[0080] Canceling the display of the game screen means that the extended reality device switches the displayed screen from the game screen to the screen of the real environment to prompt the first subject to change the movement trajectory in time to avoid collision.

[0081] In summary, the method provided by this application displays a virtual scene image when a first subject is wearing an extended reality device, and a first activity area in the physical scene is the first subject's safe activity area. When a second subject is located within the first activity area, in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirement, a prompt message or a second scene image is displayed, and the prompt message or the second scene image is used to inform the first subject of the collision relationship with the second subject. The distance between the first subject and the second subject can be used to predict a possible collision event between the first subject and the second subject, and based on the prompt message, the first subject can be promptly instructed to adjust its own motion trajectory to avoid the occurrence of a collision event and ensure the safety of the first subject.

[0082] Next, the extended reality device involved in the embodiment of the present application is described. As shown in FIG2 , FIG2 is a schematic diagram of an extended reality device provided by an exemplary embodiment of the present application. The extended reality device is the executor of the method of the present application.

[0083] Optionally, the extended reality device 200 is a head-mounted virtual reality device (also known as VR glasses). When a user wears the extended reality device 200, a virtual image appears in front of the user's eyes. The extended reality device 200 includes four camera assemblies 210, namely camera assembly C0, camera assembly C2, camera assembly C3, and camera assembly C4, located at different positions of the extended reality device 200.

[0084] Among them, the field of view angle of each camera component 210 is an obtuse angle. For example, the field of view angle of each camera component 210 is 120°, and the total field of view angle of the four camera components 210 is 480°, which exceeds 360° and can collect environmental images of the user's environment in all directions.

[0085] Among them, the extended reality device 200 also includes a control chip 220, which is used to process images, calculate the user's position, predict the user's movement trajectory, etc.

[0086] Optionally, when a first subject wears an augmented reality device, the control chip displays a virtual scene image, which is used to provide the first subject with VR scene perception; the camera assembly captures an environmental image, which is used to represent the environment in which the first subject is located; and when a second subject is within the image acquisition range of the camera assembly, the control chip displays a prompt message in response to the first subject's first predicted trajectory data and the second predicted trajectory data of the second subject meeting the prompt requirement. The first predicted trajectory data and the second predicted trajectory data are used to represent the movement trajectories of the first and second subjects at future moments.

[0087] In some embodiments, the extended reality device includes a display screen component and a control chip. When a first subject wears the extended reality device, the control chip controls the display screen component to display a virtual scene screen, and the first activity area in the physical scene is the safe activity area of ​​the first subject; when the second subject is located in the first activity area, the control chip controls the display screen component to display prompt information or a second scene screen in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirements. The prompt information or the second scene screen is used to prompt the first subject of the predicted collision relationship with the second subject.

[0088] In some embodiments, there is a communication connection relationship between the extended reality device 200 and the server. After the extended reality device 200 collects the environmental image through the camera component 210, it sends it to the server. The server performs the above-mentioned calculation processing process and returns the result information to the extended reality device 200. After receiving the result information, the extended reality device 200 displays the corresponding picture to the user.

[0089] It is worth noting that the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0090] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology, all based on the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require significant computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark and will need to be transmitted to backend systems for logical processing. Data of varying levels will be processed separately, and data from all industries will require a strong system backend, which can only be achieved through cloud computing.

[0091] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system.

[0092] In conjunction with the above-mentioned introduction to terms and application scenarios, the information prompting method based on an extended reality device provided by this application is described. This method can be executed by a server or an extended reality device, or by both the server and the extended reality device. In the embodiments of this application, the method is described using the extended reality device as an example. As shown in Figure 3, Figure 3 is a flowchart of the information prompting method based on an extended reality device provided by an exemplary embodiment of this application. The method includes the following steps.

[0093] In step 310 , a virtual scene image is displayed when the first subject wears the augmented reality device.

[0094] The first subject is in a physical scene, and the first activity area in the physical scene is a safe activity area for the first subject.

[0095] Exemplarily, the virtual scene picture is a virtual scene picture in a VR game, and the virtual scene includes virtual characters / virtual elements / virtual items, etc.

[0096] Among them, the extended reality device includes at least one of a VR device or an AR (Augmented Reality) device.

[0097] In step 320 , when the second subject is located in the first active area, in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirement, display prompt information or a second scene picture.

[0098] The prompt information or the second scene image is used to prompt the first subject of the predicted collision relationship with the second subject.

[0099] Optionally, the extended reality device includes a camera component, which is used to capture images of the physical environment in which the first subject is located.

[0100] The second scene screen refers to the image screen captured by the camera component and displayed after the extended reality device performs perspective processing on the virtual scene screen. That is, the screen presented by the extended reality device to the first subject is switched from the virtual scene screen to the real environment screen.

[0101] When a first subject wears an extended reality device, a first activity area of ​​the first subject is determined. The first activity area is used to be an environment area where the first subject performs activities, and the range divided by the first activity area is a safe activity range of the first subject.

[0102] When the first subject is within the first activity area and is performing activities, the extended reality device runs the game normally. When the first subject leaves the first activity area and is performing activities, the extended reality device stops running the game and displays a prompt message or a second scene screen to prompt the first subject to change its movement trajectory and return to the first activity area to continue activities.

[0103] For example, the first activity area is a circular activity area divided by the position of the first subject when the first subject just wears the extended reality device as the center and a preset distance as the radius. When the first subject moves within the first activity area, its position changes. If the first subject reaches the edge of the first activity area, the extended reality device immediately switches the screen, switching the virtual reality screen to the real environment screen. At this time, the prompt information is presented in the form of the screen content presented to the first subject by the extended reality device.

[0104] Optionally, the first activity area may be determined in the following ways: (1) an area automatically divided by the extended reality device for the first subject; (2) an area customized by the first subject through the extended reality device, such as when the first subject extends their arm and gestures in the air, a closed area is divided based on the trajectory of the fingertips. For another example, when the extended reality device further includes a handle portion, the first subject gestures based on the handle portion, and a closed area is divided based on the trajectory of the handle portion.

[0105] Among them, the extended reality device includes a camera component, which is used to collect environmental images, and the environmental images are used to represent the environment in which the first subject is located.

[0106] For example, the first subject is player A, and the environment in which player A is located is living room B, then the environmental image shows the situation in living room B.

[0107] In some embodiments, in order to fully display the environment in which the first subject is located, the extended reality device includes multiple camera components. The environmental image captured by each camera component shows a partial appearance of the environment. The images captured by each camera component need to be stitched together to form a complete panoramic image. This panoramic image is the environmental image, which is used for subsequent judgment and calculation.

[0108] Therefore, before determining the first activity area of ​​the first subject (or before running the VR game), the camera components need to be calibrated to solve the internal parameters of each camera component and the external parameters between each camera component.

[0109] Among them, the calibration process refers to the use of mathematical models and mathematical methods to approximate the complex mapping function between the captured image and the real world.

[0110] The main functions are as follows: (1) restore the distorted image to a normal image to facilitate subsequent image stitching; (2) transform the output of all camera components into the same coordinate system.

[0111] Exemplarily, when the extended reality device runs a game, it automatically solves the internal parameters of each camera component and the external parameters between two adjacent camera components.

[0112] They are: internal parameter K0 of camera component C0, internal parameter K1 of camera component C1, internal parameter K2 of camera component C2, internal parameter K3 of camera component C3. External parameters [R, T] between camera component C0 and camera component C1 are 01, external parameters [R, T] between camera component C1 and camera component C2 are 12, external parameters [R, T] between camera component C2 and camera component C3 are 23, and external parameters [R, T] between camera component C3 and camera component C0 are 30.

[0113] Optionally, the internal parameters of the camera assembly include, but are not limited to: (1) focal length: describes the distance between the lens and the imaging plane, which directly affects the image magnification and field of view; (2) pixel aspect ratio: describes the relative size of the pixels in the image in the horizontal and vertical directions; (3) distortion coefficient: describes the image distortion caused by external factors. Distortion can be radial (increasing with increasing distance from the image center) or tangential (caused by the non-parallelism between the lens and the image sensor plane).

[0114] Optionally, the external parameters between any two camera components are used to describe the relative position and attitude of these camera components in three-dimensional space. External parameters typically include a rotation matrix and a translation vector. The rotation matrix describes the rotation relationship from the coordinate system of one camera component to the coordinate system of another camera component. It is a 3x3 matrix that contains the rotation parameters of the three axes, which define the relative orientation between the cameras. The translation vector describes the relative displacement between the two camera components. It is a vector containing three components, corresponding to the translation parameters of the three axes. These parameters represent the position offset of one camera component relative to the other camera component in three-dimensional space.

[0115] The coordinate system of the camera component C0 is taken as the world coordinate system. The coordinate system of the camera component C0 is a three-dimensional coordinate system established with the designated reference point of the camera component C0 (for example, the designated reference point is the center position of the camera, that is, the optical center of the camera component) as the origin. The X-axis and Y-axis of the three-dimensional coordinate system are parallel to the image plane captured by the camera component C0, and the Z-axis of the three-dimensional coordinate system passes through the designated reference point and is perpendicular to the image plane captured by the camera component C0.

[0116] After obtaining the calibration results, the images captured by each camera assembly are stitched together based on the calibration results to form a complete environment image, which is a panoramic image. For example, as shown in FIG4 , FIG4 is a schematic diagram of a stitched image. The images captured by adjacent camera assemblies are placed adjacent to each other and stitched together to form a panoramic image 400.

[0117] The presence of a second subject in the first activity area is determined based on an environmental image captured by the camera assembly. The environmental image is used to represent the environment in which the first subject is located.

[0118] It is worth noting that the designated reference point of the camera assembly can be any reference point on the camera assembly, for example, the optical center of the camera assembly, the center of the lens of the camera assembly, or the designated point in the upper left corner of the lens of the camera assembly. This embodiment does not limit this.

[0119] The step of determining whether the second subject exists in the first activity area may be performed as follows: steps 510 to 540 , as shown in FIG. 5 .

[0120] Step 510: Capture an environmental image based on the camera assembly and determine whether a second subject exists in the environmental image.

[0121] Optionally, the camera component is an RGB depth camera component, and the image captured by the camera component is a color image. A first image captured by the camera component at a first moment is obtained, where the first image and the environmental image correspond to the same environmental area; pixels of the first image and pixels of the environmental image are binarized to obtain pixel data of the first image and pixel data of the environmental image; the binarized pixel data corresponds to a first numerical value or a second numerical value; based on the difference between the pixel data of the first image and the pixel data of the environmental image, the number of pixels with non-zero difference at the corresponding pixel position is counted; and in response to the number of pixels meeting a preset requirement, it is determined that a second subject exists in the environmental image.

[0122] Exemplarily, the first image captured by the camera component C0 is taken as an example for description, and the content of the first image corresponds to the local content in the panoramic image.

[0123] Among them, the extended reality device also includes a gyroscope, which can detect the posture and orientation of the subject in space. According to the offset output by the gyroscope, the picture part corresponding to the first image position can be found in the panoramic image obtained by stitching the images captured at the initial moment, and the corresponding picture part can be used as the background image.

[0124] The background image and the first image are binarized separately, that is, each pixel in the background image and the first image is assigned a value of 0 or 1 respectively, and the pixel values ​​at corresponding positions of the background image and the first image are subtracted pixel by pixel. The number of pixels M with non-zero difference values ​​is counted. If the ratio of M to the total number of pixels reaches a preset threshold, it is determined that a second subject exists in the environmental image.

[0125] Exemplarily, the first image and the background image both contain 10,000 pixels, and there is a correspondence between the pixel positions of the two images. For example, pixel 100 of the first image corresponds to pixel 100 of the background image; pixel 101 of the first image corresponds to pixel 101 of the background image.

[0126] Among them, after the binarization process, the value of pixel No. 100 of the first image is 0, the value of pixel No. 100 of the background image is 0, and the difference between the two is 0, indicating that there is no part of the second subject in the pixel; the value of pixel No. 101 of the first image is 1, the value of pixel No. 101 of the background image is 0, and the difference between the two is 1, indicating that there is part of the second subject in the pixel.

[0127] Repeating the above process for each pixel yields M = 1200, which accounts for 12% of the total number of pixels (10,000). Given a preset threshold of 10%, 12% exceeds 10%, indicating the presence of a second subject in the first image. This means that a second subject exists in the image captured by camera component C0 and in the environment.

[0128] The above steps are performed on the remaining camera components C1, C2, and C3. If a second subject exists in the image captured by any camera component, it means that a second subject exists in the environment where the first subject is located. At this time, it is necessary to further determine whether the second subject is in the first activity area.

[0129] Step 520 : In response to the presence of the second subject in the environment image, determine a first relative positional relationship between the second subject and the first subject based on the position of the second subject in the environment image.

[0130] The first relative position relationship is used to indicate the position of the second subject in the environment with respect to the first subject.

[0131] Schematically, as shown in FIG6 , FIG6 is a schematic diagram showing a first relative positional relationship between the second body and the first body.

[0132] Figure 6 is a top view. The first active area 600 is a circular area. A coordinate system is established with the center of the first active area 600 as the origin O. The position of the first subject 601 in the environment can be represented as coordinate P0. The extended reality device worn by the first subject 601 captures an image of the environment through a camera assembly. Based on the position of the second subject 602 in the image of the environment, it can be determined that, from the perspective of the first subject 601, a first relative positional relationship exists between the second subject 602 and the first subject 601: the second subject 602 is to the left and in front of the first subject 601. Furthermore, based on this first relative positional relationship and coordinate P0, the position P1 of the second subject 602 in the environment is determined.

[0133] The extended reality device includes multiple camera assemblies. Optionally, the multiple camera assemblies include adjacent first and second camera assemblies. Determining the first relative position relationship is described using environmental images captured by the first and second camera assemblies as an example.

[0134] In response to the presence of a second subject in both the first environmental image captured by the first camera component and the second environmental image captured by the second camera component, a first relative position relationship between the second subject and the first subject is determined based on the position of the second subject in the first environmental image and the position of the second subject in the second environmental image.

[0135] Exemplarily, a first extension line is drawn based on the positions of the centers of the first camera assembly and the second subject in the first environmental image, and a second extension line is drawn based on the positions of the centers of the second camera assembly and the second subject in the second environmental image; and a first relative positional relationship between the second subject and the first subject is determined based on the intersection of the first extension line and the second extension line.

[0136] Schematically, as shown in FIG7 , FIG7 is a schematic diagram of determining the first relative position relationship based on the extension line.

[0137] As shown in FIG7 , the designated reference point of the first camera assembly is O1, and the designated reference point of the second camera assembly is O2. Since both the first camera assembly and the second camera assembly belong to the extended reality device, the distance d between O1 and O2 is known.

[0138] The center of the second subject is located at point PL in first environment image 710, and the center of the second subject is located at point PR in second environment image 720. A first extension line O1PL and a second extension line O2PR are drawn. The intersection of the first and second extension lines is position P1 of the center of the second subject in the environment. Because the first subject is wearing an extended reality device and the first camera assembly and the second camera assembly are relatively close, the first relative positional relationship between the second subject and the first subject can be determined based on the intersection.

[0139] Step 530: Determine a first distance between the first body and the second body based on the first relative position relationship.

[0140] Continuing with Figure 7 , the angles of the triangle O1P2O2 can be measured, and the distance d between O1 and O2 is known. The law of cosines can be used to calculate the length of each side of the triangle O1P2O2. Because the first subject is wearing an augmented reality device and the first and second camera assemblies are relatively close, the first distance between the first and second subjects can be approximately represented by the perpendicular distance between line segments O1O2 and P1.

[0141] Step 540: Determine whether a second subject exists in the first activity area based on the position of the first subject in the environment and the first distance.

[0142] Based on the position of the first subject in the environment and the center position of the first activity area, a second relative positional relationship between the first subject and the center of the first activity area, as well as a second distance between the first subject and the center of the first activity area, is determined. The second relative positional relationship indicates the position of the first subject in the environment relative to the center of the first activity area.

[0143] As shown in Figure 6, the second relative position relationship can be represented by the coordinate P0 of the first subject 601 in the environment. The extended reality device worn by the first subject includes a gyroscope that can detect the position change of the first subject 601. The coordinate P0 can be determined by the data output by the gyroscope.

[0144] The third relative position relationship between the second subject and the center of the first active area is determined based on the first relative position relationship and the second relative position relationship. That is, in FIG. 6 , the third relative position relationship is the coordinate P1 of the second subject 602 in the environment.

[0145] A third distance between the second body and the center of the first active area is determined based on the third relative positional relationship, the first distance, and the second distance.

[0146] According to the vector formula: vector OP0+vector P0P1=vector OP1, the first distance P0P1 and the second distance OP0 are known, and the third distance OP1 can be solved.

[0147] Based on a magnitude relationship between the third distance and the area radius of the first active area, it is determined that the second subject exists in the first active area.

[0148] If the third distance is greater than the radius of the first active area, the second subject does not exist in the first active area; if the third distance is less than the radius of the first active area, the second subject exists in the first active area.

[0149] When the second subject is located in the first activity area, first motion track data of the first subject is collected, and second motion track data of the second subject is collected.

[0150] The steps of collecting the first motion trajectory data and the second motion trajectory data may be performed as follows: Steps 810 to 830 , as shown in FIG8 .

[0151] Step 810 : Continuously capture multiple frames of environmental images when the second subject is located in the first activity area.

[0152] Wherein, multiple frames of environmental images all contain the second subject.

[0153] Optionally, a world coordinate system is established with a designated reference point of the first active area as an origin; and a camera coordinate system corresponding to the extended reality device of the first subject is established.

[0154] That is, when the first subject just puts on the extended reality device to divide the first activity area, a world coordinate system is established with the designated reference point of the first activity area as the origin. At this time, the game is not running. When dividing the first activity area, the position of the center of the first subject in the environment is approximately the same as the position of the designated reference point of the first activity area in the environment. Therefore, the two-dimensional coordinate system shown in Figure 7 is actually one of the planes of the world coordinate system in step 330.

[0155] Optionally, the designated reference point may be any point in the first active area, and description will be given by taking the designated reference point as the center of the first active area as an example.

[0156] At this point, the center of the first subject and the designated reference point (center) of the first active area are the same point (coinciding), serving as the origin of the three-dimensional coordinate system. Furthermore, since the first subject and the camera assembly are also positioned approximately identically in the environment, at the initial moment (i.e., when the first subject first puts on the extended reality device), the center of the first subject, the center of the camera assembly, and the center of the first active area are approximately collinear, and this line is perpendicular to the floor of the environment.

[0157] The camera coordinate system is established with the position of the camera assembly corresponding to the first subject's XR device as the origin. This camera coordinate system is actually based on the first subject, and its origin changes as the first subject moves. The origin of the world coordinate system is a fixed point.

[0158] Optionally, the multiple camera components of the extended reality device include at least one of a grayscale camera component and an RGB depth camera component, wherein the environment image captured by the grayscale camera component is a grayscale image, and the environment image captured by the RGB depth camera component is a color image.

[0159] That is, multiple camera components can all be grayscale camera components, or all be RGB depth camera components, or some can be grayscale camera components and some can be RGB depth camera components.

[0160] For example, taking the first camera component as an RGB depth camera component, after capturing an environmental image, the first camera component inputs the environmental image into a pre-trained deep neural network (DNN) to calculate the depth information of the environmental image. The depth information of the environmental image includes the number of bits used to store each pixel. The depth information of a two-dimensional image can represent the three-dimensional information of the subject in the image, and the depth information can be used to map the subject from two-dimensional space to three-dimensional space.

[0161] For example, assuming that both the first and second camera assemblies are grayscale camera assemblies, after the adjacent first and second camera assemblies capture grayscale images, depth information is calculated based on the difference between portions of the two grayscale images belonging to the same scene. For example, since images captured by grayscale camera assemblies only provide brightness information, stereo matching can be performed using the grayscale images to determine the correspondence between the grayscale images captured by the two camera assemblies and obtain depth information.

[0162] Step 820 : Map the position of the center of the second subject in the multi-frame environment image to a three-dimensional coordinate system to obtain second motion trajectory data of the second subject.

[0163] The position of the center of the second subject in the multi-frame environmental image can be determined based on the camera coordinate system corresponding to the extended reality device worn by the first subject. Optionally, the extended reality device includes a gyroscope, which can output the position change of the first subject. The initial position of the first subject is the same as the origin position of the world coordinate system, which means that the initial coordinates of the first subject are known. When the first subject moves, the real-time coordinates of the first subject in the world coordinate system can be determined based on the position change of the first subject output by the gyroscope. After the camera component captures the multi-frame environmental image, it can determine the coordinates of the center of the second subject in the camera coordinate system based on the position of the center of the second subject in the multi-frame environmental image. At this time, the origin of the camera coordinate system is the position point of the first subject. Based on the coordinates of the first subject in the world coordinate system, the coordinates of the center of the second subject in the world coordinate system can be indirectly determined.

[0164] Exemplarily, the deep neural network pre-trained in the above step 810 is still used. The pre-trained deep neural network will frame the second subject in the environmental image using a 2D Box (Two-Dimension Box). As shown in Figure 9, the second subject 910 of the environmental image 900 is framed by a two-dimensional rectangular frame 920. At this time, the side length information of the rectangular frame 920 (width is w, height is h) and the offset (cx, cy) of the rectangular frame 920 relative to the center of the environmental image 900 can be determined. cx refers to the offset of the rectangular frame 920 in the X-axis direction, and cy refers to the offset of the rectangular frame 920 on the Y-axis.

[0165] To better predict the object's trajectory, we need to calculate the three-dimensional information of the second subject. That is, we need to use a 3D box (three-dimensional box) to frame the second subject. The center of the 3D box can more accurately represent the coordinates of the second subject in the camera coordinate system, and thus determine the position of the second subject in the world coordinate system.

[0166] The following assumptions are made: (1) The second subject rotates only along the axis perpendicular to the ground, and does not rotate along the other two axes. That is, only the Yaw offset angle exists, and the Pitch and Roll angles are both 0. The Yaw angle is used to describe the action or state of the second subject rotating around its vertical axis in the horizontal plane; the Pitch angle and Roll angle are used to describe the rotation angles of the second subject around the other two axes of the coordinate system respectively; the Yaw, Pitch, and Roll angles are parameters describing the rotation of the second subject. (2) The height of the center of the second subject is equivalent to the designated reference point (center) of the camera assembly.

[0167] Based on the above assumptions, referring to FIG. 9 , the second subject 910 of the environmental image 900 is framed by a three-dimensional frame 930. At this point, the side length information of the three-dimensional frame 930 (length l, width w, height h), the offset of the three-dimensional frame 930 relative to the center of the environmental image 900 (cx, cy, cz), and the rotation parameters (α, β, Γ) of the three-dimensional frame 930 on each axis of the three-dimensional coordinate system can be determined. Here, cx refers to the offset of the three-dimensional frame 930 in the X-axis direction, cy refers to the offset of the three-dimensional frame 930 in the Y-axis direction, and cz refers to the offset of the three-dimensional frame 930 in the Z-axis direction; α refers to the rotation angle of the three-dimensional frame 930 in the X-axis direction, β refers to the rotation angle of the three-dimensional frame 930 in the Y-axis direction, and Γ refers to the rotation angle of the three-dimensional frame 930 in the Z-axis direction.

[0168] Based on the above information, the coordinates of the three-dimensional center of the second body 910 can be obtained at the same time.

[0169] The above steps are repeated for the collected multiple frames of environmental images to obtain the coordinate change trajectory of the three-dimensional center position of the second subject 910 as the second motion trajectory data of the second subject.

[0170] Step 830: Collect first motion trajectory data of the first subject.

[0171] The extended reality device includes a gyroscope, which is used to collect first motion trajectory data of the first subject.

[0172] Optionally, in response to the extended reality device being in the first operating mode, first motion trajectory data of the first subject is collected based on a gyroscope.

[0173] The first operating mode refers to an operating mode in which the first subject wears the extended reality device and moves around, that is, the first subject is experiencing a mobile game.

[0174] Among them, when the first subject moves to the edge of the first activity area, the extended reality device will display a prompt message or a second scene picture to prompt the first subject to return to the first activity area to perform activities.

[0175] Alternatively, in response to the extended reality device being in a second operating mode, a second activity area of ​​the first subject is determined, and the first motion trajectory data is obtained. The second operating mode refers to the operating mode in which the first subject is wearing the extended reality device and performing an in-place activity, i.e., the second subject is playing an in-place activity game. The second activity area is the environment in which the first subject is performing the in-place activity, and the second activity area is smaller than the first activity area.

[0176] Exemplarily, as shown in Figure 10, the second active area 1020 is a circular area, the first active area 1010 is also a circular area, the center of the second active area 1020 coincides with the center of the first active area 1010, and the radius of the second active area 1020 is smaller than the radius of the first active area 1010.

[0177] First predicted trajectory data is obtained based on the first motion trajectory data, and second predicted trajectory data is obtained based on the second motion trajectory data, wherein the first predicted trajectory data and the second predicted trajectory data are used to represent the motion trajectories of the first subject and the second subject at a future moment.

[0178] That is, the first predicted trajectory corresponds to the first predicted trajectory data, and the second predicted trajectory corresponds to the second predicted trajectory data. The first predicted trajectory data is used to indicate the motion trajectory of the first subject in the future time period, and the second predicted trajectory data is used to indicate the motion trajectory of the second subject in the future time period.

[0179] The process of determining the first predicted trajectory data and the second predicted trajectory data is the same, and the process of determining the second predicted trajectory data is taken as an example for description.

[0180] The second motion trajectory data is input into a pre-trained machine learning network to predict the position coordinates of the second subject at multiple future moments; the position coordinates corresponding to the multiple future moments are fitted into a curve as the second predicted trajectory data.

[0181] Schematically, as shown in Figure 11, Figure 11 is a schematic diagram of the second predicted trajectory data. Figure 11 is a top view, with multiple origins representing the changes in the position of the second subject on the ground, and the horizontal axis coordinate representing the time when the second subject arrives at that position. The second predicted trajectory data is the fitted curve 1100 in Figure 11. The first half of curve 1100 is the collected second motion trajectory data, and the second half is the second predicted trajectory data predicted by the machine learning network.

[0182] The second predicted trajectory data includes the position coordinates of the second subject at multiple moments in the future time period. When predicting the trajectory, the machine learning network will correct the motion trajectory in real time.

[0183] When the first predicted trajectory data and the second predicted trajectory data meet the prompt requirement, prompt information or a second scene screen is displayed.

[0184] The prompt information is used to instruct the first subject to adjust its own motion trajectory.

[0185] Optionally, the first predicted trajectory data and the second predicted trajectory data are curve data, the first predicted trajectory corresponds to a first curve, and the second predicted trajectory corresponds to a second curve.

[0186] When there is an intersection between the first curve and the second curve, it means that the motion trajectories of the first subject and the second subject in the future time period may overlap. As shown in Figure 10, the motion trajectories of the first subject and the second subject in the future time period are represented by arrowed lines, and there is an intersection between the trajectories.

[0187] Optionally, the first predicted trajectory data and the second predicted trajectory data meet the prompt requirements, which means that there is an intersection between the first curve and the second curve, and the time interval between the moment when the first subject arrives at the intersection position and the moment when the second subject arrives at the intersection position is less than a preset time length threshold.

[0188] When there is an intersection between the first curve and the second curve, a first time when the first subject reaches the intersection and a second time when the second subject reaches the intersection are determined. In response to the time interval between the first time and the second time being less than a preset time threshold, a prompt message or a second scene image is displayed, wherein the prompt message is perceived by the first subject.

[0189] Optionally, the form of presentation of the prompt information includes but is not limited to: (1) the extended reality device initiates an alarm sound; (2) the extended reality device displays a text prompt information, such as: "Out of safe range, please return"; (3) the extended reality device directly switches the image presented to the first subject from the virtual reality image to the real environment image, that is, the real environment image is the prompt information.

[0190] In some embodiments, in addition to the above-mentioned camera assembly, gyroscope, and control chip, the extended reality device also includes a position sensor. The extended reality device including the position sensor can support multiple people to play VR games in the same space, and display prompt information or a second scene screen in time when the player moves to avoid collisions between players.

[0191] Optionally, the extended reality device worn by the first subject includes a first position sensor, the extended reality device worn by the second subject includes a second position sensor, and the second subject corresponds to the second activity area.

[0192] When a first subject wears a first extended reality device, a first activity area of ​​the first subject is determined by the first extended reality device. The first activity area is the environment area when the first subject performs activities. Wherein, there is a second subject in the environment where the first subject is located, and the second subject wears a second extended reality device, and the second subject corresponds to a second activity area.

[0193] The first activity area and the second activity area are different areas within the same environment. The first subject plays a VR game in the first activity area, and the second subject plays a VR game in the second activity area. The first position sensor and the second position sensor are in communication with each other and can transmit position information to each other.

[0194] The first position sensor of the first extended reality device receives position information transmitted by the second position sensor of the second extended reality device, thereby determining the relative positional relationship between the first subject and the second subject. The second position sensor transmits the position information of the second subject to the first position sensor. After the first position sensor receives the position information of the second subject, the extended reality device can determine the relative positional relationship between the first subject and the second subject based on the position information of the first subject collected by the first position sensor.

[0195] If the relative position relationship between the first subject and the second subject meets the prompt requirement, a prompt message or a second scene image is displayed, wherein the prompt message is used to instruct the first subject to adjust its own motion trajectory. The prompt requirement includes that the distance between the first subject and the second subject is less than or equal to a preset distance threshold.

[0196] In summary, the method provided by this application displays a virtual scene image when a first subject is wearing an extended reality device, and a first activity area in the physical scene is the first subject's safe activity area. When a second subject is located within the first activity area, in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirement, a prompt message or a second scene image is displayed, and the prompt message or the second scene image is used to inform the first subject of the collision relationship with the second subject. The distance between the first subject and the second subject can be used to predict a possible collision event between the first subject and the second subject, and based on the prompt message, the first subject can be promptly instructed to adjust its own motion trajectory to avoid the occurrence of a collision event and ensure the safety of the first subject.

[0197] FIG12 is a structural block diagram of an information prompting apparatus based on an extended reality device provided by an exemplary embodiment of the present application. As shown in FIG12 , the apparatus includes the following parts.

[0198] The scene screen display module 1210 is configured to display a virtual scene screen when a first subject wears the augmented reality device, wherein the first activity area in the physical scene is a safe activity area for the first subject;

[0199] The prompt information display module 1220 is used to display prompt information or a second scene picture when the second subject is located in the first activity area, in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirements. The prompt information or the second scene picture is used to prompt the first subject of the collision relationship with the second subject.

[0200] In an optional embodiment, the extended reality device includes a camera assembly;

[0201] The prompt information display module 1220 further includes:

[0202] An activity area determining unit 1221 is configured to determine the first activity area of ​​the first subject, where the first activity area is an environment area where the first subject is active;

[0203] A subject determining unit 1222 is configured to determine the presence of the second subject in the first activity area based on an environmental image captured by the camera assembly, wherein the environmental image is used to represent the environment in which the first subject is located;

[0204] The collecting unit 1223 is configured to collect first motion trajectory data of the first subject and second motion trajectory data of the second subject when the second subject is located in the first activity area;

[0205] a prediction unit 1224 configured to predict first predicted trajectory data based on the first motion trajectory data, and to predict second predicted trajectory data based on the second motion trajectory data, wherein the first predicted trajectory data and the second predicted trajectory data are used to represent motion trajectories of the first subject and the second subject at a future moment;

[0206] The prompt information display unit 1225 is configured to display the prompt information or the second scene image when the first predicted trajectory data and the second predicted trajectory data meet the prompt requirement.

[0207] In an optional embodiment, the subject determination unit 1222 is also used to determine, in response to the presence of the second subject in the environmental image, a first relative position relationship between the second subject and the first subject based on the position of the second subject in the environmental image, the first relative position relationship being used to indicate the position of the second subject in the environment with the first subject as a reference; determine a first distance between the first subject and the second subject based on the first relative position relationship; and determine the presence of the second subject in the first activity area based on the position of the first subject in the environment and the first distance.

[0208] In an optional embodiment, the extended reality device includes a plurality of camera components, wherein the plurality of camera components include a first camera component and a second camera component that are adjacent to each other;

[0209] The subject determination unit 1222 is further configured to determine, in response to the presence of the second subject in both the first environment image captured by the first camera assembly and the second environment image captured by the second camera assembly, the first relative positional relationship between the second subject and the first subject based on the position of the second subject in the first environment image and the position of the second subject in the second environment image;

[0210] Determining, based on the position of the first subject in the environment and the center position of the first activity area, a second relative positional relationship between the first subject and the center of the first activity area, and a second distance between the first subject and the center of the first activity area, wherein the second relative positional relationship indicates the position of the first subject in the environment relative to the center of the first activity area;

[0211] determining a third relative positional relationship between the second subject and the center of the first active area based on the first relative positional relationship and the second relative positional relationship;

[0212] determining a third distance between the second body and the center of the first active area based on the third relative positional relationship, the first distance, and the second distance;

[0213] Based on a size relationship between the third distance and an area radius of the first activity area, it is determined that the second subject exists in the first activity area.

[0214] In an optional embodiment, the subject determination unit 1222 is further used to make a first extension line based on the positions of the centers of the first camera assembly and the second subject in the first environmental image, and to make a second extension line based on the positions of the centers of the second camera assembly and the second subject in the second environmental image; and to determine the first relative position relationship between the second subject and the first subject based on the intersection of the first extension line and the second extension line.

[0215] In an optional embodiment, the camera assembly is an RGB depth camera assembly;

[0216] The subject determination unit 1222 is also used to obtain a first image captured by the camera component at a first moment, where the first image and the environmental image correspond to the same environmental area; binarize the pixels of the first image and the pixels of the environmental image to obtain pixel data of the first image and pixel data of the environmental image; the binarized pixel data corresponds to a first value or a second value; based on the difference between the pixel data of the first image and the pixel data of the environmental image, count the number of pixels with non-zero difference at the corresponding pixel position; in response to the number of pixels meeting the preset requirements, determine that the second subject exists in the environmental image.

[0217] In an optional embodiment, the extended reality device includes a gyroscope;

[0218] The collecting unit 1223 is further configured to collect the first motion trajectory data of the first subject based on the gyroscope in response to the extended reality device being in a first operating mode, where the first operating mode refers to an operating mode in which the first subject moves while wearing the extended reality device; or, in response to the extended reality device being in a second operating mode, determine an in-place activity area of ​​the first subject to obtain the first motion trajectory data, where the second operating mode refers to an operating mode in which the first subject performs an in-place activity while wearing the extended reality device, where the in-place activity area is an environment area in which the first subject performs the in-place activity, and the in-place activity area is smaller than the first activity area.

[0219] The prediction unit 1224 is further configured to input the second motion trajectory data into a pre-trained machine learning network to predict the position coordinates of the second subject at multiple future moments; and fit the position coordinates corresponding to the multiple future moments into a curve as the second predicted trajectory data.

[0220] In an optional embodiment, the first predicted trajectory corresponds to a first curve, and the second predicted trajectory corresponds to a second curve;

[0221] The prompt information display module 1220 is also used to determine the first moment when the first subject reaches the intersection and the second moment when the second subject reaches the intersection when there is an intersection between the first curve and the second curve; in response to the time interval between the first moment and the second moment being less than a preset duration threshold, display the prompt information or the second scene picture.

[0222] In an optional embodiment, before the subject determination unit 1222, as shown in FIG13, the prompt information display module 1220 further includes:

[0223] A coordinate system establishing unit 1226 is configured to establish a world coordinate system with the designated reference point of the first activity area as the origin; and establish a camera coordinate system corresponding to the extended reality device of the first subject;

[0224] The prompt information display module 1220 is also used to continuously capture multiple frames of environmental images when the second subject is located in the first activity area, and the multiple frames of environmental images all contain the second subject; map the position of the second subject in the multiple frames of environmental images to the world coordinate system through the camera coordinate system to obtain the second position of the second subject corresponding to the first activity area; and display the prompt information or the second scene picture when the distance between the first position and the second position meets the prompt requirements.

[0225] In an optional embodiment, the extended reality device includes a first position sensor, the extended reality device worn by the second subject includes a second position sensor, and the second subject corresponds to the second activity area;

[0226] The prompt information display module 1220 is also used to receive the position information sent by the second position sensor through the first position sensor, and determine the relative position relationship between the first subject and the second subject; when the relative position relationship between the first subject and the second subject meets the prompt requirements, the prompt information or the second scene picture is displayed; wherein the prompt requirements include that the distance between the first subject and the second subject is less than or equal to a preset distance threshold.

[0227] In summary, the device provided by the present application displays a virtual scene screen when a first subject wears an augmented reality device, wherein the first activity area in the physical scene is the first subject's safe activity area; and when a second subject is within the first activity area, in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirement, a prompt message or a second scene screen is displayed, wherein the prompt message or the second scene screen is used to inform the first subject of the collision relationship with the second subject. The device can predict a possible collision event between the first subject and the second subject based on the distance between the first subject and the second subject, and timely instruct the first subject to adjust its own motion trajectory based on the prompt message to avoid the occurrence of a collision event and ensure the safety of the first subject.

[0228] It should be noted that the aforementioned embodiments of the information prompt device based on an extended reality device are merely illustrative of the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, i.e., the internal structure of the device can be divided into different functional modules to perform all or part of the functions described above. Furthermore, the aforementioned embodiments of the information prompt device based on an extended reality device and the embodiments of the information prompt method based on an extended reality device are based on the same concept. The specific implementation process is detailed in the method embodiments and will not be further described here.

[0229] In some embodiments, the method proposed in this application can be executed by an extended reality device as well as by a computer device as shown in FIG14 , wherein a communication connection relationship exists between the computer device and the extended reality device. The computer device 1400 can be: a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. The computer device 1400 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0230] Typically, the computer device 1400 includes a processor 1401 and a memory 1402 .

[0231] The processor 1401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1401 may be a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA

[0232] (Programmable Logic Array, programmable logic array) is implemented in at least one hardware form. The processor 1401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0233] Memory 1402 may include one or more computer-readable storage media, which may be non-transitory. Memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 1402 is used to store at least one instruction, which is executed by processor 1401 to implement the information prompt method based on the extended reality device provided in the method embodiment of the present application.

[0234] In some embodiments, the computer device 1400 also includes other components. Those skilled in the art will understand that the structure shown in Figure 14 does not constitute a limitation on the computer device 1400, and it may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0235] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0236] An embodiment of the present application also provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the information prompt method based on an extended reality device as described in any of the above embodiments of the present application.

[0237] An embodiment of the present application also provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the information prompt method based on an extended reality device as described in any of the above embodiments of the present application.

[0238] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the information prompting method based on an extended reality device described in any of the above embodiments.

[0239] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0240] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An information prompting method based on an extended reality device, characterized in that: The method comprises: When a first subject wears the extended reality device, a virtual scene image is displayed, and a first activity area in the physical scene is a safe activity area for the first subject; When the second subject is located within the first activity area, in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirements, a prompt message or a second scene picture is displayed, and the prompt message or the second scene picture is used to prompt the first subject of the collision relationship with the second subject.

2. The method according to claim 1, characterized in that The extended reality device includes a camera component; When the second subject is located in the first activity area, in response to a distance between the first position of the first subject and the second position of the second subject meeting a prompt requirement, displaying prompt information or a second scene picture includes: determining the first activity area of ​​the first subject; determining the presence of the second subject within the first activity area based on an environmental image captured by the camera assembly, the environmental image being used to represent the environment in which the first subject is located; When a second subject is located within the first activity area, collecting first motion trajectory data of the first subject, and collecting second motion trajectory data of the second subject; Predicting first predicted trajectory data based on the first motion trajectory data, and predicting second predicted trajectory data based on the second motion trajectory data, wherein the first predicted trajectory data and the second predicted trajectory data are used to represent the motion trajectories of the first subject and the second subject at a future moment; When the first predicted trajectory data and the second predicted trajectory data meet the prompt requirement, the prompt information or the second scene picture is displayed.

3. The method according to claim 2, characterized in that The determining, based on the environment image captured by the camera assembly, that the second subject exists in the first activity area includes: In response to the second subject being present in the environment image, determining a first relative positional relationship between the second subject and the first subject based on the position of the second subject in the environment image, the first relative positional relationship being used to indicate a position of the second subject in the environment relative to the first subject; determining a first distance between the first subject and the second subject based on the first relative position relationship; It is determined that the second subject exists in the first activity area based on the position of the first subject in the environment and the first distance.

4. The method according to claim 3, characterized in that The extended reality device includes a plurality of camera components, wherein the plurality of camera components include a first camera component and a second camera component that are adjacent to each other; In response to the second subject existing in the environment image, determining a first relative positional relationship between the second subject and the first subject based on a position of the second subject in the environment image includes: In response to the second subject being present in both a first environment image captured by the first camera assembly and a second environment image captured by the second camera assembly, determining the first relative positional relationship between the second subject and the first subject based on a position of the second subject in the first environment image and a position of the second subject in the second environment image; The determining, based on the position of the first subject in the environment and the first distance, that the second subject exists in the first activity area includes: Determining, based on the position of the first subject in the environment and the center position of the first activity area, a second relative positional relationship between the first subject and the center of the first activity area, and a second distance between the first subject and the center of the first activity area, wherein the second relative positional relationship indicates the position of the first subject in the environment relative to the center of the first activity area; determining a third relative positional relationship between the second subject and the center of the first active area based on the first relative positional relationship and the second relative positional relationship; determining a third distance between the second body and the center of the first active area based on the third relative positional relationship, the first distance, and the second distance; Based on a size relationship between the third distance and an area radius of the first activity area, it is determined that the second subject exists in the first activity area.

5. The method according to claim 4, characterized in that The determining the first relative positional relationship between the second subject and the first subject based on the position of the second subject in the first environment image and the position of the second subject in the second environment image includes: Draw a first extension line based on the positions of the centers of the first camera assembly and the second body in the first environment image, and draw a second extension line based on the positions of the centers of the second camera assembly and the second body in the second environment image; The first relative positional relationship between the second body and the first body is determined based on an intersection of the first extension line and the second extension line.

6. The method according to claim 3, characterized in that The camera component is an RGB depth camera component; Before determining, in response to the second subject existing in the environment image, a first relative positional relationship between the second subject and the first subject based on the position of the second subject in the environment image, the method further includes: Acquire a first image captured by the camera assembly at a first moment, where the first image and the environmental image correspond to the same environmental area; Binarizing pixels of the first image and pixels of the environment image to obtain pixel data of the first image and pixel data of the environment image; the pixel data after the binarization processing corresponds to a first value or a second value; Based on the difference between the pixel data of the first image and the pixel data of the environment image, counting the number of pixels at corresponding pixel positions whose difference is non-zero; In response to the number of pixels meeting a preset requirement, it is determined that the second subject exists in the environment image.

7. The method according to claim 2, characterized in that The extended reality device includes a gyroscope; The collecting the first motion trajectory data of the first subject includes: In response to the extended reality device being in a first operating mode, collecting the first motion trajectory data of the first subject based on the gyroscope, the first operating mode being an operating mode when the first subject wears the extended reality device to move; or In response to the extended reality device being in a second operating mode, determining an in-place activity area of ​​the first subject, and obtaining the first motion trajectory data, wherein the second operating mode is an operating mode when the first subject wears the extended reality device and performs an in-place activity, and the in-place activity area is an environment area when the first subject performs the in-place activity, and the in-place activity area is smaller than the first activity area; The step of predicting and obtaining second predicted trajectory data based on the second motion trajectory data includes: Inputting the second motion trajectory data into a pre-trained machine learning network to predict the position coordinates of the second subject at multiple future moments; The position coordinates corresponding to the multiple future moments are fitted into a curve as the second predicted trajectory data.

8. The method according to claim 2, characterized in that The first predicted trajectory data corresponds to a first curve, and the second predicted trajectory data corresponds to a second curve; The displaying of prompt information or a second scene image in response to the distance between the first position of the first subject and the second position of the second subject meeting a prompt requirement includes: If there is an intersection between the first curve and the second curve, determining a first time when the first subject reaches the intersection and a second time when the second subject reaches the intersection; In response to the time interval between the first moment and the second moment being less than a preset time threshold, the prompt information or the second scene picture is displayed.

9. The method according to claim 2, characterized in that Before determining that the second subject exists in the first activity area based on the environmental image collected by the camera assembly, the method further includes: Establishing a world coordinate system with the designated reference point of the first activity area as the origin; establishing a camera coordinate system corresponding to the extended reality device of the first subject; When the second subject is located in the first activity area, in response to a distance between the first position of the first subject and the second position of the second subject meeting a prompt requirement, displaying prompt information or a second scene picture includes: When the second subject is located in the first activity area, continuously capturing multiple frames of environmental images, each of which includes the second subject; Mapping the position of the second subject in the multi-frame environment image to the world coordinate system through the camera coordinate system to obtain the second position of the second subject corresponding to the first activity area; When the distance between the first position and the second position meets the prompt requirement, the prompt information or the second scene picture is displayed.

10. The method according to claim 1, characterized in that The extended reality device includes a first position sensor, the extended reality device worn by the second subject includes a second position sensor, and the second subject corresponds to a second activity area; When the second subject is located in the first activity area, in response to a distance between the first position of the first subject and the second position of the second subject meeting a prompt requirement, displaying prompt information or a second scene picture includes: receiving, by the first position sensor, position information sent by the second position sensor, and determining a relative positional relationship between the first body and the second body; When the relative position relationship between the first subject and the second subject meets the prompt requirements, the prompt information or the second scene picture is displayed; wherein the prompt requirements include that the distance between the first subject and the second subject is less than or equal to a preset distance threshold.

11. An extended reality device, characterized in that: The extended reality device includes a display screen component and a control chip; The control chip is configured to control the display screen assembly to display a virtual scene image when a first subject wears the augmented reality device, wherein the first activity area in the physical scene is a safe activity area for the first subject; The control chip is also used to control the display screen assembly to display prompt information or a second scene picture in response to the distance between the first position of the first body and the second position of the second body meeting the prompt requirement when the second body is located in the first active area. The prompt information or the second scene picture is used to prompt the first body of the collision relationship with the second body.

12. An information prompting device based on an extended reality device, characterized in that: The extended reality device includes a camera component, and the device includes: A scene screen display module, configured to display a virtual scene screen when a first subject wears the augmented reality device, wherein the first activity area in the physical scene is a safe activity area for the first subject; A prompt information display module is used to display prompt information or a second scene picture when the second subject is located in the first activity area, in response to the distance between the first position of the first subject and the second position of the second subject meeting the prompt requirements. The prompt information or the second scene picture is used to prompt the first subject of the collision relationship with the second subject.

13. A computer-readable storage medium, characterized in that The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the information prompt method based on the extended reality device as described in claims 1 to 10.

14. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the information prompting method based on an extended reality device as described in claims 1 to 10.

Images