Image processing method, device, equipment and storage medium
By acquiring and converting the position of the headset and handles, the equipment needs for users' wear and venues are reduced, and the problems of high wear complexity, high carrying weight and high interaction cost in existing multi-person interaction solutions are solved, achieving better user experience and scope of activity.
Patent Information
- Application Number
- CN202010607638.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-11-26
AI Technical Summary
In the existing multi-person interaction solutions, users have high wear complexity, large carrying weight, limited range of movement and high interaction costs, mainly due to the need for multiple optical tracking Mark points, tracking sensors, PC computers and tracking camera servers.
By obtaining the position of the headset and handle, determine the location of the virtual character's local world coordinate system, and converting it to the reference world coordinate system and sending it to other headsets, realizing the rendering of the virtual scene and reducing the devices worn by users. and venue requirements.
It improves the user experience and range of activities, reduces the cost of interaction, and realizes real-time observation of other wearers' behavior in virtual scenes.
Smart Images

Figure CN111966213B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and more specifically, to an image processing method, an image processing device, a head mounted device and a computer readable storage medium. Background Art
[0002] At present, the interaction of multiple people in a virtual scene (referred to as multi-person interaction) through VR (Virtual Reality) equipment has become one of the important technologies in virtual reality technology.
[0003] When multiple people are interacting, it is critical to achieve a 1:1 restoration of the body and hand movements of the user wearing the VR device in the virtual scene. Figure 1 As shown, each user is usually required to wear a VR head-mounted display device and hold a handle controller in each hand (the handle controller can be a gun, a sword, or a bow and arrow in the virtual scene, etc.). Optical tracking Mark points are placed on the joints of each active part of the user's legs, arms, etc., and on the handle controller, or tracking sensors (IMU inertial navigation unit, electromagnetic sensor, etc.) are worn. At the same time, the user is required to carry a small PC, and set up multiple tracking cameras and a camera server in the interactive venue. When the user moves freely in the venue space, the tracking camera will track the optical Mark points on various parts of the user's body and the handle controller, as well as the user's VR head-mounted display device in real time, and transmit the tracking data obtained in real time to the camera server. The camera server calculates the tracking data of all users and transmits it to the PC carried by each user in real time through the data cable. The PC combines the transmitted user tracking data to render the scene, and then displays it in real time on the VR head-mounted display device of the corresponding user. In this way, users can see each other's virtual actions in the virtual scene through the VR head-mounted display device, including the user's walking, various movements of the handle controller, whether the user is swinging his arm, etc.
[0004] However, existing multi-person interaction solutions have the following problems: multiple optical tracking Mark points or tracking sensors are placed on the user's body, which increases the complexity of the user's wearing; each user needs to carry a PC, and the increased carrying weight and heat dissipation of the computer reduce the user experience; each user needs to be connected to the tracking camera server through a cable, which limits the user's range of activities; in addition to the VR head-mounted display device, multiple Mark points and multiple other devices need to be set up, such as tracking cameras, tracking camera servers, PCs, etc., which increases the interaction cost. Summary of the invention
[0005] An object of the present invention is to provide a new image processing scheme.
[0006] According to a first aspect of the present invention, there is provided an image processing method, which is applied to any head mounted device in an interactive system, comprising:
[0007] Obtaining the positions of the head mounted device, a handle matched with the head mounted device, and feet of a wearer wearing the head mounted device in a local world coordinate system of the head mounted device;
[0008] Determine, according to the acquired position, the position of the virtual character matching the wearer in the local world coordinate system;
[0009] Convert the position of the virtual character in the local world coordinate system to a position in a reference world coordinate system, and send the position to other head mounted devices in the interactive system;
[0010] Receiving the position of the corresponding virtual character in the reference world coordinate system from other head mounted devices in the interactive system;
[0011] The virtual scene is rendered according to the position of the corresponding virtual character in the reference coordinate system.
[0012] Optionally, obtaining the positions of the feet of the wearer in a local world coordinate system of the head mounted device includes:
[0013] According to the position of the head mounted device in the local world coordinate system, the positions of both feet of a wearer wearing the head mounted device in the local world coordinate system are estimated.
[0014] Optionally, obtaining a position of a handle matched with the head mounted device in a local world coordinate system of the head mounted device includes:
[0015] Determine the relative position between the handle and the head mounted device according to an electromagnetic signal emitted by a handle matched with the head mounted device and inertial data of the handle;
[0016] The position of the handle in the local world coordinate system is determined according to the relative position and the position of the head mounted device in the local world coordinate system.
[0017] Optionally, converting the position of the virtual character in the local world coordinate system to a position in a reference world coordinate system includes:
[0018] Using the first conversion relationship, the second conversion relationship, and the third conversion relationship, the position of the virtual character in the local world coordinate system is converted to the position in the reference world coordinate system;
[0019] Among them, the first conversion relationship is the conversion relationship between the local world coordinate system of the head-mounted device and the local camera coordinate system of the head-mounted device, the second conversion relationship is the conversion relationship between the local camera coordinate system of the head-mounted device and the reference camera coordinate system, and the third conversion relationship is the conversion relationship between the reference camera coordinate system and the reference world coordinate system.
[0020] Optionally, the method further includes a step of acquiring the second conversion relationship, including:
[0021] Acquire a feature vector of each first feature point in a historical frame image collected by the head mounted device and a feature vector of each second feature point in a set space image collected by the reference device, the historical frame image belonging to the set space image;
[0022] Determine a first preset number of feature point pairs according to the feature vector of the first feature point and the feature vector of the second feature point, wherein a pair of the feature point pairs consists of one of the first feature points and a matching second feature point;
[0023] The second conversion relationship is determined according to the position of the first feature point in each feature point pair in the local camera coordinate system of the head mounted device and the position of the corresponding second feature point in the reference camera coordinate system.
[0024] Optionally, the method further includes:
[0025] acquiring an image of the wearer's hand;
[0026] Determine, based on the hand image, positions of the skeleton points of the wearer's hands in the local world coordinate system;
[0027] The virtual character is updated according to the position of the skeleton point in the local world coordinate system.
[0028] Optionally, the method further includes:
[0029] Acquire images of the wearer's eyes and mouth;
[0030] Determine the positions of the eyes and the mouth of the wearer in the local world coordinate system according to the images of the eyes and the mouth of the wearer;
[0031] The virtual character is updated according to the positions of the eyes and the mouth in the local world coordinate system.
[0032] According to a second aspect of the present invention, there is provided an image processing apparatus, comprising:
[0033] An acquisition module, used to acquire positions of the head mounted device, a handle matched with the head mounted device, and feet of a wearer wearing the head mounted device in a local world coordinate system of the head mounted device;
[0034] A determination module, configured to determine, based on the acquired position, a position of a virtual character matching the wearer in the local world coordinate system;
[0035] A conversion module, used to convert the position of the virtual character in the local world coordinate system to a position in a reference world coordinate system, and send the position to other head mounted devices in the interactive system;
[0036] A receiving module, used for receiving the position of the corresponding virtual character in the reference world coordinate system from other head mounted devices in the interactive system;
[0037] The rendering module is used to render the virtual scene according to the position of the corresponding virtual character in the reference coordinate system.
[0038] According to a third aspect of the present invention, there is provided a head mounted device, comprising the apparatus according to the second aspect;
[0039] Alternatively, comprising a memory and a processor, wherein:
[0040] The memory is used to store executable instructions, and the instructions are used to control the processor to execute the method according to any one of the first aspects.
[0041] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores computer instructions, and when the computer instructions in the storage medium are executed by a processor, the method as described in any one of the first aspects is implemented.
[0042] In this embodiment, the positions of the head-mounted device, the handle matched with the head-mounted device, and the feet of the wearer wearing the head-mounted device in the local world coordinate system of the head-mounted device are obtained. According to the obtained position, the position of the virtual character matched with the wearer in the local world coordinate system is determined. Based on this, the head-mounted device can obtain a virtual character with a 1:1 ratio with the wearer. The virtual character is represented by the positions of multiple feature points that can outline the outline of the virtual character in the local world coordinate system. The head-mounted device converts the position of the virtual character in the local world coordinate system to the position in the reference world coordinate system, and sends it to other head-mounted devices in the interactive system. At the same time, the head-mounted device receives the position of the corresponding virtual character in the reference world coordinate system from other head-mounted devices in the interactive system, and can obtain the position of the virtual character matched with the wearer of other head-mounted devices in the interactive system in the same coordinate system; according to the position of the corresponding virtual character in the reference coordinate system, the virtual scene is rendered. That is, the behavior of other wearers can be observed in real time in the virtual scene. In this way, multi-person interaction can be achieved. In this embodiment, the wearer only needs to wear the head-mounted device and hold the matching handle. In this way, it is avoided to place multiple optical tracking marks or tracking sensors on the wearer, carry a PC on the back, and connect to the tracking camera server with cables. This improves the wearer's experience and increases the wearer's range of activities. At the same time, the venue for multi-person interaction does not need to set up equipment such as tracking cameras, tracking camera servers, PCs, etc., which reduces the interaction cost.
[0043] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0045] Figure 1 It is a schematic diagram of the existing scenario for realizing multi-person interaction;
[0046] Figure 2 is a block diagram of a hardware configuration of a head mounted device for implementing an image processing method provided in an embodiment of the present invention;
[0047] Figure 3 is a flowchart of an image processing method provided by an embodiment of the present invention;
[0048] Figure 4 is a schematic diagram of the structure of a head mounted device provided by an embodiment of the present invention;
[0049] Figure 5is a schematic diagram of a multi-person interaction scenario implemented by the image processing method provided by an embodiment of the present invention;
[0050] Figure 6 is a schematic diagram of a marking point corresponding to a human eye provided by an embodiment of the present invention;
[0051] Figure 7 is a schematic diagram of a mouth corresponding to annotated points provided by an embodiment of the present invention;
[0052] Figure 8 is a structural schematic diagram of an image processing device provided by an embodiment of the present invention;
[0053] Fig. 9 It is a schematic structural diagram of another head mounted device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0054] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.
[0055] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0056] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0057] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0058] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0059] <Hardware Configuration Example>
[0060] Figure 2 The present invention is a block diagram of a hardware configuration of a head mounted device for implementing an image processing method according to an embodiment of the present invention.
[0061] The head mounted device 2000 may be a VR head mounted display device or an MR head mounted display device.
[0062] The head mounted device 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, a display device 2500, an input device 2600, a speaker 2700, a microphone 2800, and the like. The processor 2100 may be a central processing unit CPU, a microprocessor MCU, and the like. The memory 2200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 2300 may include, for example, a USB interface, a headphone interface, and the like. The communication device 2400 may, for example, be capable of wired or wireless communication. The display device 2500 may be, for example, a liquid crystal display screen, a touch display screen, and the like. The input device 2600 may include, for example, a touch screen, a keyboard, and the like. The user may input / output voice information through the speaker 2700 and the microphone 2800.
[0063] Despite Figure 2 In the figure, multiple devices are shown for the head mounted device 2000, however, the present invention may only involve some of the devices, for example, the head mounted device 2000 only involves the memory 2200 and the processor 2100.
[0064] In the embodiment of the present invention, the memory 2200 of the head mounted device 2000 is used to store instructions, and the instructions are used to control the processor 2100 to execute the image processing method provided by the embodiment of the present invention.
[0065] In the above description, a technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.
[0066] <Method Example>
[0067] An embodiment of the present invention provides an image processing method, which is applied to any head mounted device in an interactive system. Any head mounted device may be Figure 2 The head mounted device 2000 is shown. Also, the interactive system includes at least two head mounted devices, and the head mounted devices in the interactive system are communicatively connected with each other.
[0068] like Figure 3 As shown, the image processing method provided by the embodiment of the present invention includes the following S3100-S3500:
[0069] S3100: Obtain positions of a head mounted device, a handle matched with the head mounted device, and both feet of a wearer of the head mounted device in a local world coordinate system of the head mounted device.
[0070] In this embodiment, the position of the head mounted device in the local world coordinate system of the head mounted device is obtained as follows: Figure 4As shown, the head-mounted device is provided with four cameras C1, C2, C3 and C4 for tracking the external environment (the cameras C1, C2, C3 and C4 constitute a 6DOF module), and the field of view FOV of each camera is at least 150°, and the frame rate is above 30Hz. In addition, an inertial measurement unit IMU is provided on the head-mounted device, and the frequency of IMU information acquired by the IMU is 1000Hz. The head-mounted device obtains 6DOF information of the head-mounted device in the local world coordinate system based on the external environment scanned at the same time by the four cameras for tracking the external environment. The head-mounted device fuses the IMU information with the 6DOF information to obtain 6DOF information that meets the refresh rate of the head-mounted device. Furthermore, the head-mounted device obtains the position of the head-mounted device in the local world coordinate system based on the fused 6DOF information.
[0071] It can be understood that the head mounted device and the head of the wearer wearing the head mounted device are relatively stationary, therefore, the position of the head mounted device in the local world coordinate system can be regarded as the position of the head of the wearer wearing the head mounted device in the local world coordinate system.
[0072] It should be noted that the IMU information can also compensate for the 6DOF information of the head-mounted device in the local world coordinate system, making the 6DOF information of the head-mounted device in the local world coordinate system more accurate. In the embodiment of the present invention, other methods can also be used to obtain the position of the head-mounted device in the local world coordinate system.
[0073] In one embodiment, the method for obtaining the position of the handle matched with the head mounted device in the local world coordinate system may be as described above in the method for obtaining the position of the head mounted device in the local world coordinate system.
[0074] In another embodiment, the method for obtaining the position of the handle matched with the head mounted device in the local world coordinate system may be other. Based on this, the method for obtaining the position of the handle matched with the head mounted device in the local world coordinate system of the head mounted device includes the following S3110 and S3111:
[0075] S3110. Determine the relative position between the handle and the head mounted device based on the electromagnetic signal emitted by the handle matched with the head mounted device and the inertial data of the handle.
[0076] In this embodiment, for each of the handles matched with the head-mounted device, an electromagnetic transmitter and an IMU are provided in the handle. The electromagnetic transmitter includes three mutually perpendicular electromagnetic coils in three-dimensional directions. An electromagnetic receiver is provided in the head-mounted device. The electromagnetic transmitter provided in the handle transmits an electromagnetic signal of a certain frequency. The electromagnetic receiver in the head-mounted device receives the electromagnetic signal transmitted by the electromagnetic transmitter provided in the handle. The head-mounted device processes the received electromagnetic signal to obtain 6DOF information of the handle relative to the electromagnetic receiver in the head-mounted device. In addition, the head-mounted device obtains the IMU information in the handle through the built-in wireless transmission module of the head-mounted device, that is, inertial data, to compensate for the 6DOF information of the handle relative to the electromagnetic receiver in the head-mounted device, so as to obtain more accurate 6DOF information of the handle relative to the electromagnetic receiver in the head-mounted device.
[0077] Furthermore, based on the conversion relationship between the electromagnetic receiver in the head mounted device and the local world coordinate system of the head mounted device, the 6DOF information of the handle relative to the head mounted device can be obtained. Furthermore, based on the 6DOF information of the handle relative to the head mounted device, the head mounted device can obtain the relative position between the handle and the head mounted device.
[0078] S3111. Determine the position of the handle in the local world coordinate system based on the relative position and the position of the head mounted device in the local world coordinate system.
[0079] In an embodiment, based on the position of the head mounted device in the local world coordinate system, the relative position obtained by the above S3110 is superimposed to obtain the position of the handle in the local world coordinate system.
[0080] It can be understood that the handle and the hand of the wearer wearing the head-mounted device are relatively stationary. Therefore, the position of the handle in the local world coordinate system can be regarded as the position of the corresponding hand of the wearer wearing the head-mounted device in the local world coordinate system.
[0081] In this embodiment, the position of the feet of the wearer wearing the head mounted device in the local world coordinate system of the head mounted device may be acquired in the following manner S3220:
[0082] S3220: Estimate the positions of both feet of a wearer of the head mounted device in the local world coordinate system according to the position of the head mounted device in the local world coordinate system.
[0083] In this embodiment, since the movement of the feet can drive the head to move, there is a correlation between the movement of the feet and the movement of the head. Based on this, the movement of the feet can be estimated based on the movement of the head. The estimation can be performed using a motion estimation engine, such as the estimation method used in Unity3D or Unreal.
[0084] It should be noted that, in the embodiment of the present invention, other methods may be used to obtain the positions of the feet of the wearer wearing the head mounted device in the local world coordinate system.
[0085] S3200: Determine the position of the virtual character matching the wearer in the local world coordinate system according to the acquired position.
[0086] In this embodiment, the head mounted device fuses the positions obtained based on the above S3100 to obtain a virtual character that is 1:1 with the wearer. It can be understood that the virtual character is represented by the positions of multiple feature points that can outline the virtual character in the local world coordinate system.
[0087] S3300: Convert the position of the virtual character in the local world coordinate system to the position in the reference world coordinate system, and send it to other head-mounted devices in the interactive system.
[0088] In this embodiment, the reference world coordinate system is the world coordinate system of the reference device in the interactive system. The reference device in the interactive system can be specified by the user and can be a default head mounted device.
[0089] In this embodiment, when the head mounted device used to perform the image processing method provided by the embodiment of the present invention is the reference device, the position of the virtual character in the local world coordinate system is the position of the virtual character in the reference world coordinate system. The position of the virtual character in the local world coordinate system refers to the coordinate position of the feature points constituting the virtual character in the local world coordinate system.
[0090] In one embodiment, the user inputs an instruction to the head mounted device, and the head mounted device determines whether it is a reference device according to the instruction. In this embodiment, the specific content of the input instruction is not limited.
[0091] In another embodiment, the head mounted device stores an identification of whether it is a reference device, and the head mounted device can determine whether it is a reference device based on the identification.
[0092] In the case where the head mounted device that performs the image processing method provided by the embodiment of the present invention is not a reference device, the conversion in S3300 may be implemented as follows 3310:
[0093] S3310. Use the first conversion relationship, the second conversion relationship, and the third conversion relationship to convert the position of the virtual character in the local world coordinate system to the position in the reference world coordinate system.
[0094] Among them, the first transformation relationship is the transformation relationship between the local world coordinate system of the head-mounted device and the local camera coordinate system of the head-mounted device, the second transformation relationship is the transformation relationship between the local camera coordinate system of the head-mounted device and the reference camera coordinate system, and the third transformation relationship is the transformation relationship between the reference camera coordinate system and the reference world coordinate system.
[0095] In this embodiment, the reference camera coordinate system is the camera coordinate system of the reference device. Through the first conversion relationship, the position of the virtual character in the local world coordinate system can be converted to the position in the local camera coordinate system of the head-mounted device. Further, through the second conversion relationship, the position of the virtual character in the local camera coordinate system of the head-mounted device can be converted to the position in the reference camera coordinate system. Further, through the third conversion relationship, the position of the virtual character in the reference camera coordinate system can be converted to the position in the reference world coordinate system. In this way, the local world coordinate system of the head-mounted device can be converted to the reference world coordinate system, thereby realizing the display of the virtual character in the unified world coordinate system of the reference device.
[0096] S3400. Receive the position of the corresponding virtual character in the reference world coordinate system from other head-mounted devices in the interactive system.
[0097] In this embodiment, other head mounted devices in the interactive system simultaneously perform the above steps S3100-S3300, so that the head mounted devices that perform the image processing method provided by the embodiment of the present invention can obtain the positions of the virtual characters matched by the wearers of other head mounted devices in the interactive system in the reference coordinate system. That is, the positions of the virtual characters matched by the wearers of other head mounted devices in the interactive system in the same coordinate system can be obtained.
[0098] S3500: Render a virtual scene according to the position of the corresponding virtual character in the reference coordinate system.
[0099] In the embodiment of the present invention, the head mounted device renders the virtual scene based on the received position of the virtual character in the reference coordinate system, and can display multiple wearers' 1:1 virtual characters. That is, the behavior of other wearers can be observed in the virtual scene in real time. In this way, multi-person interaction can be achieved.
[0100] In one example, corresponding to Figure 1 The game scene shown in FIG. 1 is processed by the image processing method provided in this embodiment. Figure 5 As shown, users can achieve multi-person interaction by only wearing a head-mounted device and holding a simulated gun.
[0101] In this embodiment, the positions of the head-mounted device, the handle matched with the head-mounted device, and the feet of the wearer wearing the head-mounted device in the local world coordinate system of the head-mounted device are obtained. According to the obtained position, the position of the virtual character matched with the wearer in the local world coordinate system is determined. Based on this, the head-mounted device can obtain a virtual character with a 1:1 ratio with the wearer. The virtual character is represented by the positions of multiple feature points that can outline the outline of the virtual character in the local world coordinate system. The head-mounted device converts the position of the virtual character in the local world coordinate system to the position in the reference world coordinate system, and sends it to other head-mounted devices in the interactive system. At the same time, the head-mounted device receives the position of the corresponding virtual character in the reference world coordinate system from other head-mounted devices in the interactive system, and can obtain the position of the virtual character matched with the wearer of other head-mounted devices in the interactive system in the same coordinate system; according to the position of the corresponding virtual character in the reference coordinate system, the virtual scene is rendered. That is, the behavior of other wearers can be observed in real time in the virtual scene. In this way, multi-person interaction can be achieved. In this embodiment, the wearer only needs to wear the head-mounted device and hold the matching handle. In this way, it is avoided to place multiple optical tracking marks or tracking sensors on the wearer, carry a PC on the back, and connect to the tracking camera server with cables. This improves the wearer's experience and increases the wearer's range of activities. At the same time, the venue for multi-person interaction does not need to set up equipment such as tracking cameras, tracking camera servers, PCs, etc., which reduces the interaction cost.
[0102] On the basis of the above S3310, the image processing method provided by the embodiment of the present invention further includes the step of obtaining a first conversion relationship. This step includes the following S3311 and S3312:
[0103] S3311. For the historical frame images collected by the head mounted device, obtain the positions of a second preset number of first feature points in the image coordinate system of the historical frame images and their positions in the local world coordinate system of the head mounted device.
[0104] S3312: Determine a first conversion relationship according to the positions of a second preset number of first feature points in the image coordinate system of the historical frame image and their positions in the local world coordinate system of the head mounted device.
[0105] In an embodiment of the present invention, the head mounted device collects images in advance and records them as historical frame images. The head mounted device then extracts the first feature points in the historical frame images and arbitrarily selects a second preset number of first feature points from the extracted first feature points. At the same time, the head mounted device obtains the position of each first feature point in the second preset number of first feature points in the world coordinate system of the head mounted device based on the 6DOF module provided therein.
[0106] At the same time, for each extracted first feature point and the corresponding historical frame image, the position of each first feature point in the image coordinate system of the corresponding historical frame image can be obtained.
[0107] Based on the positions of the second preset number of first feature points obtained above in the image coordinate system of the historical frame image, and their positions in the local world coordinate system of the head-mounted device, the PNP algorithm can be used to obtain the conversion relationship between the local camera coordinate system of the head-mounted device and the local world coordinate system of the head-mounted device, that is, the first conversion relationship mentioned above.
[0108] It should be noted that the above-mentioned second preset number can be set based on experience.
[0109] On the basis of the above S3310, the image processing method provided by the embodiment of the present invention further includes the step of obtaining a second conversion relationship. This step includes the following S3313-S3315:
[0110] S3313. Obtain a feature vector of each first feature point in a historical frame image collected by the head mounted device and a feature vector of each second feature point in a set space image collected by the reference device, where the historical frame image belongs to the set space image.
[0111] Among them, the historical frame image belongs to the set space image.
[0112] In this embodiment, the set space image is a multi-frame image obtained by scanning the entire set space by the reference device. The set space refers to the space where the interactive system operates. For example, a user plays a game in a room through the interactive system. Then the set space is the room. The historical frame image collected by the head-mounted device is an image collected by the head-mounted device at a certain position in the set space.
[0113] Based on the above S3313, it can be known that the image processing method provided by the embodiment of the present invention also includes a step of determining a feature vector of each first feature point in the historical frame image collected by the head-mounted device. This step can be implemented by the following S3313-1:
[0114] S3313-1. In the historical frame image, determine a feature vector corresponding to the first feature point according to the grayscale value of the pixel of the first feature point and the grayscale values of the pixels in the neighborhood of the first feature point.
[0115] In one embodiment, the neighborhood of the first feature point may be a 5*5 window area centered on the first feature point, or other areas, which are not limited in this embodiment.
[0116] For each first feature point in the historical image frame, the difference between the grayscale value of each pixel in the neighborhood and the grayscale value of the first feature point is calculated; the maximum difference among all the differences obtained is extracted; each difference obtained is normalized using the maximum difference, and the feature vector corresponding to the first feature point is obtained.
[0117] At the same time, based on the above S3313, it can be known that the image processing method provided by the embodiment of the present invention also includes the step of obtaining the feature vector of each second feature point in the set space image collected by the reference device. In this embodiment, the feature vector of each second feature point in the set space image collected by the reference device is calculated by the reference device and sent to the head-mounted device. Among them, sending includes sending directly from the reference device to the head-mounted device, or the reference device uploading the feature vector of each second feature point in the set space image collected by itself to the cloud; based on this, the head-mounted device then obtains it from the cloud.
[0118] S3314. Determine a first preset number of feature point pairs based on the feature vector of the first feature point and the feature vector of the second feature point, wherein a pair of feature point pairs consists of a first feature point and a matching second feature point.
[0119] A pair of feature points consists of a first feature point and a matching second feature point.
[0120] In this embodiment, feature point matching is performed on the feature vector of each first feature point in the historical frame image collected by the head-mounted device and the feature vector of each second feature point in the set space image collected by the reference device to obtain the most matching first preset number of feature point pairs. It can be understood that the first feature point and the second feature point in a pair of feature points are the same point in space.
[0121] The first preset number may be set based on experience. In one embodiment, the first preset number may be set to 50. Also, the feature point matching may be performed using any of the matching methods such as Hamming distance matching, KNN matching, and RANSAC matching.
[0122] S3315. Determine a second transformation relationship according to the position of the first feature point in each feature point pair in the local camera coordinate system of the head mounted device and the position of the corresponding second feature point in the reference camera coordinate system.
[0123] In this embodiment, the method for determining the position of the first feature point in each feature point pair in the local camera coordinate system of the head-mounted device is as follows: based on the position of the corresponding first feature point in the image coordinate system in the historical frame image, the position of the corresponding first feature point in the local camera coordinate system of the head-mounted device is determined by using the perspective projection relationship.
[0124] In this embodiment, for each feature point pair, the position of the first feature point in the local camera coordinate system of the head-mounted device and the corresponding position of the second feature point in the reference camera coordinate system can be used to obtain a second transformation relationship based on the camera stereo geometric imaging and perspective projection relationship.
[0125] According to the relationship between camera stereo geometry and perspective projection, the position of the first feature point in each feature point pair in the local camera coordinate system of the head mounted device and the position of the corresponding second feature point in the reference camera coordinate system have the following relationship:
[0126]
[0127] Wherein, ΔT is the conversion relationship between the reference camera coordinate system and the local camera coordinate system of the head mounted device; For each feature point pair, the position of the first feature point in the local camera coordinate system of the head mounted device; is the position of the corresponding second feature point in the reference camera coordinate system.
[0128] Based on the above formula, ΔT can be obtained. Furthermore, based on ΔT, ΔT can be obtained -1 , that is, the second conversion relationship.
[0129] On the basis of the above S3310, the image processing method provided by the embodiment of the present invention further includes the step of obtaining a third conversion relationship. This step includes the following S3316 and S3317:
[0130] S3316. For the set space image captured by the reference device, obtain positions of a third preset number of second feature points in the image coordinate system and in the reference world coordinate system.
[0131] S3317. Determine a third conversion relationship according to the positions of a third preset number of second feature points in the image coordinate system and in the reference world coordinate system.
[0132] In this embodiment, the third preset number can be set based on experience. Also, the position of the second feature point in the image coordinate system and the position in the reference world coordinate system in S3316 are calculated by the reference device and sent to the head-mounted device. The sending includes sending directly from the reference device to the head-mounted device; or the reference device uploads the position of the second feature point in the image coordinate system and the position in the reference world coordinate system in S3316 to the cloud; based on this, the head-mounted device obtains it from the cloud.
[0133] In this embodiment, the specific implementation of the above S3317 is similar to the specific implementation of the above S3312. Therefore, this embodiment will not repeat the above S3317.
[0134] On the basis of any of the above embodiments, the image processing method provided by the embodiment of the present invention further includes the following S3510-S3512:
[0135] S3510. Acquire images of the wearer's eyes and mouth.
[0136] In an embodiment, Figure 4 As shown, camera devices C5 and C6 are arranged at the bottom of the left eye display screen and the right eye display screen of the head mounted device. The camera device C5 is used to obtain the image of the wearer's left eye, and the camera device C6 is used to obtain the image of the wearer's right eye.
[0137] A camera C7 for capturing the wearer's mouth image is arranged near the nose tip contact position of the head mounted display device. The camera C7 is used to obtain the wearer's mouth image.
[0138] S3511. Determine the positions of the eyes and mouth in the local world coordinate system based on the images of the wearer's eyes and mouth.
[0139] In this embodiment, the implementation of the above S3511 may be:
[0140] S3511-1. Use the traditional Zhang Zhengyou calibration method to obtain the internal parameters of the corresponding camera devices for the above C5, C6, and C7, respectively, which are recorded as: K1[3*3], K2[3*3], and K3[3*3].
[0141] S3511-2. Collect human eye samples. The human eye samples include human eye images of different ages, different genders, wearing glasses and not wearing glasses. A human eye image includes a left eye image and a right eye image. Each left eye image and right eye image in each human eye sample are respectively annotated with 16 annotation points. The 16 annotation points can outline the shape of the human eye. Use the human eye samples to train an eye tracking model. Among them, Figure 6 As shown, the 16 marking points of the left eye include 10 marking points that can outline the eyeball of the left eye and 6 marking points that can outline the shape of the left eye. Correspondingly, the 16 marking points of the right eye include 10 marking points that can outline the eyeball of the right eye and 6 marking points that can outline the shape of the right eye.
[0142] S3511-3, input the binocular image obtained based on the above S3510 into the eye tracking model trained based on the above S3511-2, and obtain 32 annotated points of the left eye and the right eye corresponding to the binocular image obtained in the above S3510. Further, obtain the image coordinates of the 32 annotated points in the binocular image obtained in the above S3510.
[0143] S3511-4, normalize the image coordinates of the 32 labeled points in the image of both eyes obtained in S3510.
[0144] Specifically, for the 16 marked points of the left eye, determine the average coordinates of the image coordinates of the 10 marked points corresponding to the eyeball of the left eye in the image of the two eyes obtained in the above S3510. Subtract each of the 16 marked points of the left eye from the corresponding average value to obtain the normalized positions of the 16 marked points of the left eye.
[0145] Similarly, for the 16 marking points of the right eye, the normalization method of the 16 marking points of the left eye is also adopted to obtain the normalized positions of the 16 marking points of the right eye.
[0146] S3511-5. Collect mouth samples. The mouth samples include mouth images of different shapes (such as open mouth, closed mouth, pouting mouth, etc.), different genders, with or without beards, and different beard shapes. In each mouth sample, 10 annotation points of the mouth shape are marked. Using the mouth samples, the mouth tracking model is trained. Figure 7 As described above, the 10 annotation points in each mouth sample can outline the shape of the mouth.
[0147] S3511-6, input the mouth image obtained based on the above S3510 into the mouth tracking model obtained based on the above S3511-5, and obtain 10 annotation points corresponding to the mouth shape in the mouth image. Further, obtain the image coordinates of the 10 annotation points in the mouth image obtained in the above S3510.
[0148] S3511-7. Normalize the image coordinates of the 10 annotation points in the image of the mouth obtained in the above S3510.
[0149] Specifically, the average value of the image coordinates of the 10 obtained annotation points in the image of the mouth obtained in S3510 is determined. The image coordinates of each of the 10 obtained annotation points in the image of the mouth obtained in S3510 are subtracted from the average value to obtain the normalized positions of the 10 annotation points of the mouth.
[0150] S3511-8. Based on the normalized image coordinates of the 42 annotation points of the left eye, right eye and mouth on the corresponding image, the camera imaging perspective principle and perspective relationship, obtain the corresponding three-dimensional position in the world coordinate system of the head-mounted device.
[0151] Specifically, the above S3511-8 is implemented by the following formula:
[0152]
[0153] Among them, K is the internal parameter of the camera device corresponding to the corresponding annotation point; (x, y) is the normalized image coordinate of the corresponding annotation point; (X, Y, Z) is the three-dimensional position of the corresponding annotation point in the world coordinate system of the head-mounted device. z is the average distance between the camera and the face, which can be set according to actual experience. For example, z = 15cm.
[0154] It should be noted that the normalized image coordinates of the 42 marking points of the left eye, the right eye and the mouth on the corresponding image can also be filtered through the image coordinate system of the corresponding marking points in the historical frame image. In this way, the shaking of the marking points can be prevented.
[0155] S3512. Update the virtual character according to the positions of the eyes and the mouth in the local world coordinate system.
[0156] In this embodiment, the virtual character is updated according to the positions of the eyes and mouth in the local world coordinate system, so that the eyes and mouth of the virtual character are consistent with the eyes and mouth of the wearer. Since the expression of the human face is mainly reflected by the mouth and eyes, based on the above S3512, the virtual character has the same facial expression as the wearer.
[0157] In this embodiment, the virtual character has the same facial expression as the wearer, so that in multi-person interaction, each wearer can understand the facial expressions of other wearers, which improves the wearer's immersion in the virtual scene.
[0158] On the basis of any of the above embodiments, the image processing method provided by the embodiment of the present invention further includes the following S3610-S3612:
[0159] S3610. Acquire an image of the wearer's hand.
[0160] In this embodiment, since the wearer holds the handle to perform a series of interactions during multi-person interaction, the wearer's hands will appear within the field of view of any one of the four camera devices for tracking the external environment set on the head mounted device. Based on this, the image of the external environment acquired by any one of the four camera devices for tracking the external environment set on the head mounted device is used as the image of the wearer's hands.
[0161] S3611. Determine the positions of the skeleton points of the wearer's hands in the local world coordinate system based on the hand image.
[0162] In this embodiment, the hand detection model is first trained. Based on this, after the hand image is input into the trained hand detection model, the area where the hand is located is output.
[0163] Then, the two-dimensional skeleton point detection model is trained. Based on this, after the image of the hand area is input into the trained two-dimensional skeleton point detection model, the position of the hand skeleton point in the hand image is obtained.
[0164] Finally, based on the positions of the hand skeleton points in the hand image, the positions of the hand skeleton points in the local world coordinate system are obtained.
[0165] S3612. Update the virtual character according to the position of the skeleton point in the local world coordinate system.
[0166] In this embodiment, the virtual character is updated according to the position of the skeleton point in the local world coordinate system so that the gestures of the virtual character's hands are consistent with the gestures of the wearer. Therefore, based on the above S3612, the virtual character has gestures consistent with the wearer.
[0167] In this embodiment, the virtual character has gestures consistent with the wearer, so that in multi-person interaction, each wearer can understand the gestures of other wearers, based on which a bare-hand interaction scenario can be realized, while also improving the wearer's sense of immersion in the virtual scene.
[0168] <Device Example>
[0169] like Figure 8 As shown, the embodiment of the present invention further provides an image processing device 80. The device 80 includes an acquisition module 81, a determination module 82, a conversion module 83, a receiving module 84 and a rendering module 85. Among them:
[0170] An acquisition module 81 is used to acquire positions of the head mounted device, a handle matched with the head mounted device, and feet of a wearer wearing the head mounted device in a local world coordinate system of the head mounted device;
[0171] A determination module 82, configured to determine the position of the virtual character matching the wearer in the local world coordinate system according to the acquired position;
[0172] A conversion module 83, used for converting the position of the virtual character in the local world coordinate system to a position in a reference world coordinate system, and sending the position to other head mounted devices in the interactive system;
[0173] A receiving module 84, configured to receive the position of the corresponding virtual character in the reference world coordinate system from other head mounted devices in the interactive system;
[0174] The rendering module 85 is used to render the virtual scene according to the position of the corresponding virtual character in the reference coordinate system.
[0175] In one embodiment, the acquisition module 81 is specifically used to estimate the positions of both feet of a wearer wearing the head mounted device in the local world coordinate system according to the position of the head mounted device in the local world coordinate system.
[0176] In one embodiment, the acquisition module 81 is specifically used to: determine the relative position between the handle and the head mounted device according to the electromagnetic signal emitted by the handle matched with the head mounted device and the inertial data of the handle;
[0177] The position of the handle in the local world coordinate system is determined according to the relative position and the position of the head mounted device in the local world coordinate system.
[0178] In one embodiment, the conversion module 83 is specifically used to: convert the position of the virtual character in the local world coordinate system to the position in the reference world coordinate system by using the first conversion relationship, the second conversion relationship, and the third conversion relationship;
[0179] Among them, the first conversion relationship is the conversion relationship between the local world coordinate system of the head-mounted device and the local camera coordinate system of the head-mounted device, the second conversion relationship is the conversion relationship between the local camera coordinate system of the head-mounted device and the reference camera coordinate system, and the third conversion relationship is the conversion relationship between the reference camera coordinate system and the reference world coordinate system.
[0180] In one embodiment, the conversion module 83 further includes an acquisition unit, which is used to:
[0181] Acquire a feature vector of each first feature point in a historical frame image collected by the head mounted device and a feature vector of each second feature point in a set space image collected by the reference device, the historical frame image belonging to the set space image;
[0182] Determine a first preset number of feature point pairs according to the feature vector of the first feature point and the feature vector of the second feature point, wherein a pair of the feature point pairs consists of one of the first feature points and a matching second feature point;
[0183] The second conversion relationship is determined according to the position of the first feature point in each feature point pair in the local camera coordinate system of the head mounted device and the position of the corresponding second feature point in the reference camera coordinate system.
[0184] In one embodiment, the image processing device 80 further includes a first updating module, and the first updating module is further configured to:
[0185] acquiring an image of the wearer's hand;
[0186] Determine, based on the hand image, positions of the skeleton points of the wearer's hands in the local world coordinate system;
[0187] The virtual character is updated according to the position of the skeleton point in the local world coordinate system.
[0188] In one embodiment, the image processing device 80 further includes a second updating module, and the second updating module is further configured to:
[0189] Acquire images of the wearer's eyes and mouth;
[0190] Determine the positions of the eyes and the mouth of the wearer in the local world coordinate system according to the images of the eyes and the mouth of the wearer;
[0191] The virtual character is updated according to the positions of the eyes and the mouth in the local world coordinate system.
[0192] <Equipment Embodiment>
[0193] like Fig. 9 As shown, the embodiment of the present invention also provides a head mounted device 90, including Figure 8 The image processing device 80 is shown.
[0194] Or, including a memory 91 and a processor 92; wherein:
[0195] The memory 91 is used to store executable instructions, and the instructions are used to control the processor 92 to execute the image processing method according to any one of the above method embodiments.
[0196] In this embodiment, the head mounted device 92 may be a VR head mounted display device or an MR head mounted display device.
[0197] <Storage Medium Embodiment>
[0198] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores computer instructions. When the computer instructions in the storage medium are executed by a processor, an image processing method as described in any one of the above method embodiments is implemented.
[0199] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0200] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0201] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0202] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.
[0203] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.
[0204] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0205] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0206] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.
[0207] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.
Claims
1. An image processing method, It is characterized in that Applicable to any head mounted device in the interactive system, including: Obtaining the positions of the head mounted device, a handle matched with the head mounted device, and feet of a wearer wearing the head mounted device in a local world coordinate system of the head mounted device; Determine, according to the acquired position, the position of the virtual character matching the wearer in the local world coordinate system; The position of the virtual character in the local world coordinate system is converted to a position in a reference world coordinate system, and sent to other head-mounted devices in the interactive system; the position in the reference world coordinate system is obtained by converting the position of the virtual character in the local world coordinate system by using a first conversion relationship, a second conversion relationship, and a third conversion relationship; wherein the first conversion relationship is a conversion relationship between a local world coordinate system of the head-mounted device and a local camera coordinate system of the head-mounted device, the second conversion relationship is a conversion relationship between a local camera coordinate system of the head-mounted device and a reference camera coordinate system, and the third conversion relationship is a conversion relationship between the reference camera coordinate system and the reference world coordinate system; Receiving the position of the corresponding virtual character in the reference world coordinate system from other head mounted devices in the interactive system; The virtual scene is rendered according to the position of the corresponding virtual character in the reference world coordinate system.
2. The method according to claim 1, It is characterized in that The obtaining the positions of the feet of the wearer in the local world coordinate system of the head mounted device includes: According to the position of the head mounted device in the local world coordinate system, the positions of both feet of a wearer wearing the head mounted device in the local world coordinate system are estimated.
3. The method according to claim 1, It is characterized in that The obtaining the position of the handle matched with the head mounted device in the local world coordinate system of the head mounted device includes: Determine the relative position between the handle and the head mounted device according to an electromagnetic signal emitted by a handle matched with the head mounted device and inertial data of the handle; The position of the handle in the local world coordinate system is determined according to the relative position and the position of the head mounted device in the local world coordinate system.
4. The method according to claim 1, It is characterized in that The method further comprises the step of acquiring the second conversion relationship, comprising: Obtaining a feature vector of each first feature point in a historical frame image collected by the head mounted device and a feature vector of each second feature point in a set space image collected by a reference device, wherein the historical frame image belongs to the set space image; Determine a first preset number of feature point pairs according to the feature vector of the first feature point and the feature vector of the second feature point, wherein a pair of the feature point pairs consists of one of the first feature points and a matching second feature point; The second conversion relationship is determined according to the position of the first feature point in each feature point pair in the local camera coordinate system of the head mounted device and the position of the corresponding second feature point in the reference camera coordinate system.
5. The method according to claim 1, It is characterized in that The method further comprises: acquiring an image of the wearer's hand; Determine, based on the hand image, positions of the skeleton points of the wearer's hands in the local world coordinate system; The virtual character is updated according to the position of the skeleton point in the local world coordinate system.
6. The method according to claim 1, It is characterized in that The method further comprises: Acquire images of the wearer's eyes and mouth; Determine the positions of the eyes and the mouth of the wearer in the local world coordinate system according to the images of the eyes and the mouth of the wearer; The virtual character is updated according to the positions of the eyes and the mouth in the local world coordinate system.
7. An image processing device, It is characterized in that include: An acquisition module, used to acquire positions of a head mounted device, a handle matched with the head mounted device, and feet of a wearer wearing the head mounted device in a local world coordinate system of the head mounted device; A determination module, configured to determine, based on the acquired position, a position of a virtual character matching the wearer in the local world coordinate system; A conversion module, used to convert the position of the virtual character in the local world coordinate system to a position in a reference world coordinate system, and send the position to other head mounted devices in the interactive system; The position in the reference world coordinate system is obtained by transforming the position of the virtual character in the local world coordinate system by using a first transformation relationship, a second transformation relationship, and a third transformation relationship; wherein the first transformation relationship is a transformation relationship between the local world coordinate system of the head-mounted device and the local camera coordinate system of the head-mounted device, the second transformation relationship is a transformation relationship between the local camera coordinate system of the head-mounted device and the reference camera coordinate system, and the third transformation relationship is a transformation relationship between the reference camera coordinate system and the reference world coordinate system; A receiving module, used for receiving the position of the corresponding virtual character in the reference world coordinate system from other head mounted devices in the interactive system; The rendering module is used to render the virtual scene according to the position of the corresponding virtual character in the reference world coordinate system.
8. A head mounted device, It is characterized in that comprising the device as claimed in claim 7; Alternatively, comprising a memory and a processor, wherein: The memory is used to store executable instructions, and the instructions are used to control the processor to execute the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, It is characterized in that The storage medium stores computer instructions. When the computer instructions in the storage medium are executed by a processor, the method according to any one of claims 1 to 6 is implemented.
