Parameter determination method and apparatus, and electronic device and computer-readable storage medium

Abstract

A parameter determination method and apparatus, and an electronic device and a computer-readable storage medium. A split-type device comprises a first sub-device and a second sub-device which are in communication connection, wherein the second sub-device comprises an audio output device and an image collection apparatus. The parameter determination method comprises: acquiring an image collected by an image collection apparatus in the direction of a first sub-device; on the basis of the image, determining a relative pose of the first sub-device and a second sub-device; and on the basis of the relative pose, determining an operating parameter of an audio output device.

Description

Parameter determination method, device, electronic device and computer-readable storage medium

[0001] This application claims priority to Chinese patent application No. 202410007854.6 filed on January 2, 2024, the entire text of which is incorporated by reference as a part of this application. Technical Field

[0002] Embodiments of the present disclosure relate to a parameter determination method for a split-type device, a parameter determination apparatus for a split-type device, an electronic device, and a computer-readable storage medium. Background Art

[0003] With the rapid development of science and technology, technologies such as VR (Virtual Reality) and AR (Augmented Reality) are increasingly attracting people to try and use them. These technologies are new audio-visual technologies based on computers. Integrating relevant scientific and technological advances, they create digital environments that closely resemble the real world in terms of vision, hearing, and touch, within a certain range. Users, with the necessary equipment, interact with objects in the digital environment, influencing each other and gaining a sense and experience that approximates the real world. This is achieved through display devices, tracking and positioning equipment, tactile interaction devices, data acquisition equipment, and specialized chips.

[0004] Summary of the Invention

[0005] At least one embodiment of the present disclosure provides a parameter determination method for a split-type device, wherein the split-type device includes a first sub-device and a second sub-device that are communicatively connected, and the second sub-device includes an audio output device and an image acquisition device. The method includes: acquiring an image captured by the image acquisition device in the direction of the first sub-device; determining a relative position between the first sub-device and the second sub-device based on the image; and determining operating parameters of the audio output device based on the relative position.

[0006] For example, in the parameter determination method provided in at least one example of the above-mentioned embodiments of the present disclosure, the relative posture of the first sub-device and the second sub-device is determined based on the image, including: inputting the feature information corresponding to the image into a posture recognition model to determine the conversion parameters between the first coordinate system and the second coordinate system constructed based on the physical points of the image acquisition device based on the posture recognition model, the conversion parameters are used to represent the relative posture of the first sub-device and the second sub-device, wherein the first coordinate system is constructed based on the first sub-device or the wearing part of the first sub-device.

[0007] For example, in the parameter determination method provided in at least one example of the above-mentioned embodiments of the present disclosure, the first sub-device includes a plurality of marker points, the image contains at least some of the plurality of marker points, and the method further includes: extracting geometric features of at least some of the marker points from the image, wherein the geometric features include at least one of the center of mass, slope, and convex hull area; wherein the feature information corresponding to the image includes the geometric features.

[0008] For example, in the parameter determination method provided by at least one example of the above-mentioned embodiments of the present disclosure, the feature information corresponding to the image includes pixel information of all or part of the pixels of the image.

[0009] For example, in the parameter determination method provided by at least one example of the above-mentioned embodiments of the present disclosure, the second sub-device includes multiple image acquisition devices, wherein the feature information corresponding to the image is input into the posture recognition model to obtain the conversion parameters between the first coordinate system and the second coordinate system based on the posture recognition model, including: inputting the feature information corresponding to the images captured by at least some of the multiple image acquisition devices into the posture recognition model separately to obtain the candidate conversion parameters corresponding to the at least some of the image acquisition devices; processing the candidate conversion parameters corresponding to the at least some of the image acquisition devices to obtain the conversion parameters.

[0010] For example, in the parameter determination method provided by at least one example of the above-mentioned embodiments of the present disclosure, the first sub-device includes a plurality of marker points, and the image contains at least some of the marker points among the plurality of marker points, wherein the candidate conversion parameters corresponding to the at least some of the image acquisition devices are processed to obtain the conversion parameters, including: determining the measurement coordinates of the marker points in the captured image for each of the at least some of the image acquisition devices, determining the reprojection coordinates of the marker points based on the corresponding candidate conversion parameters, and taking the deviation between the measurement coordinates and the reprojection coordinates as the error of the candidate conversion parameters; and selecting a candidate conversion parameter with the smallest error among the candidate conversion parameters corresponding to the at least some of the image acquisition devices as the conversion parameter.

[0011] For example, in the parameter determination method provided by at least one example of the above-mentioned embodiments of the present disclosure, the second sub-device includes multiple image acquisition devices, wherein the feature information corresponding to the image is input into the posture recognition model to obtain the conversion parameters between the first coordinate system and the second coordinate system based on the posture recognition model, including: inputting the features corresponding to the images acquired by the multiple image acquisition devices into the posture recognition model together to obtain the conversion parameters corresponding to the multiple image acquisition devices.

[0012] For example, in the parameter determination method provided in at least one example of the above embodiments of the present disclosure, determining the operating parameters of the audio output device based on the relative posture includes: determining the coordinate values ​​of the audio output device in the first coordinate system based on the conversion parameters; and determining the operating parameters of the audio output device based on the coordinate values ​​of the audio output device in the first coordinate system.

[0013] For example, in the parameter determination method provided in at least one example of the above embodiments of the present disclosure, determining the coordinate value of the audio output device in the first coordinate system based on the conversion parameter includes: determining the coordinate value of the audio output device in the first coordinate system based on the conversion parameter and the coordinate of the audio output device in the second coordinate system.

[0014] For example, in the parameter determination method provided in at least one example of the above-mentioned embodiments of the present disclosure, determining the coordinate value of the audio output device in the first coordinate system based on the conversion parameter includes: determining the coordinate measurement value of the audio output device in the first coordinate system at the first moment based on the conversion parameter determined at the first moment and the coordinate of the audio output device in the second coordinate system; determining the coordinate prediction value of the audio output device in the first coordinate system at the first moment based on historical coordinate measurement values ​​obtained within a period of time before the first moment, wherein the historical coordinate measurement value is the coordinate measurement value of the audio output device in the first coordinate system obtained based on images obtained within a period of time before the first moment; and fusing the coordinate measurement value with the coordinate prediction value to obtain a fused coordinate value, and using the fused coordinate value as the coordinate value of the audio output device in the first coordinate system at the first moment.

[0015] For example, in the parameter determination method provided in at least one example of the above-mentioned embodiments of the present disclosure, it also includes: determining the coordinate prediction value of the audio output device in the first coordinate system at the first moment based on the historical coordinate data before the first moment, and using the coordinate prediction value as the coordinate value of the audio output device in the first coordinate system at the first moment.

[0016] For example, in the parameter determination method provided in at least one example of the above-mentioned embodiments of the present disclosure, determining the coordinate prediction value of the audio output device in the first coordinate system at the first moment based on the historical coordinate data before the first moment includes: updating the parameters of the Kalman filter based on the historical coordinate data before the first moment to obtain the Kalman filter after the parameter update; and determining the coordinate prediction value of the audio output device in the first coordinate system at the first moment using the Kalman filter after the parameter update.

[0017] For example, in the parameter determination method provided in at least one example of the above-mentioned embodiments of the present disclosure, determining the operating parameters of the audio output device based on the coordinate value of the audio output device in the first coordinate system includes: when the first coordinate system is constructed based on the wearing position of the first sub-device, determining the coordinate value of the virtual sound source point on the first sub-device in the first coordinate system based on the conversion parameters between the first coordinate system and a third coordinate system constructed based on the physical point of the first sub-device; and determining the operating parameters of the audio output device in the first coordinate system based on the coordinate value of the virtual sound source point and the coordinate value of the audio output device.

[0018] For example, in the parameter determination method provided in at least one example of the above-mentioned embodiments of the present disclosure, determining the operating parameters of the audio output device based on the coordinate values ​​of the audio output device in the first coordinate system includes: when the first coordinate system is constructed based on the first sub-device, determining the coordinate values ​​of a virtual sound source point on the first sub-device in the fourth coordinate system and the coordinate values ​​of the audio output device in the fourth coordinate system based on conversion parameters between the first coordinate system and a fourth coordinate system constructed based on the wearing position of the first sub-device; and determining the operating parameters of the audio output device in the fourth coordinate system based on the coordinate values ​​of the virtual sound source point and the coordinate values ​​of the audio output device.

[0019] For example, in the parameter determination method provided in at least one example of the above embodiments of the present disclosure, the wearing part of the first sub-device is the head, the wearing part of the second sub-device is the neck, and the audio output device is a speaker.

[0020] At least one embodiment of the present disclosure provides a split device, comprising: a first sub-device and a second sub-device, the second sub-device comprising an audio output device and an image acquisition device; wherein the first sub-device or the second sub-device comprises a processor, the image acquisition device is used to acquire an image in the direction of the first sub-device and send the acquired image to the processor, the processor is used to determine a relative position between the first sub-device and the second sub-device based on the image, determine operating parameters of the audio output device based on the relative position, and send the operating parameters to the audio output device.

[0021] At least one embodiment of the present disclosure provides an electronic device, comprising a processor; a memory storing one or more computer program modules; wherein the one or more computer program modules are configured to be executed by the processor to implement the parameter determination method provided by any embodiment of the present disclosure.

[0022] At least one embodiment of the present disclosure provides a computer-readable storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the parameter determination method provided by any embodiment of the present disclosure can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 shows a flow chart of a parameter determination method provided by at least one embodiment of the present disclosure;

[0024] FIG2 shows a schematic diagram of a split-type device provided by at least one embodiment of the present disclosure;

[0025] FIG3 is a schematic diagram showing an image acquisition and processing process according to at least one embodiment of the present disclosure;

[0026] FIG4 shows a flowchart of another parameter determination method provided by at least one embodiment of the present disclosure;

[0027] FIG5 shows a schematic diagram of determining sound parameters provided by at least one embodiment of the present disclosure;

[0028] FIG6 shows a schematic block diagram of a parameter determination device provided by at least one embodiment of the present disclosure;

[0029] FIG7 shows a schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure;

[0030] FIG8 shows a schematic block diagram of another electronic device provided by at least one embodiment of the present disclosure;

[0031] FIG9 shows a schematic diagram of a computer-readable storage medium provided by at least one embodiment of the present disclosure;

[0032] FIG10 is a schematic diagram of an application scenario in which a positioning device includes an inertial measurement unit according to at least one embodiment of the present disclosure;

[0033] FIG11 is a schematic diagram of an application scenario in which a positioning device provided according to at least one embodiment of the present disclosure includes two inertial measurement units;

[0034] FIG12 is a schematic diagram showing the positions of a binocular camera and an infrared light source according to at least one embodiment of the present disclosure;

[0035] FIG13 is a schematic diagram of a main structural block diagram of a split-type wearable device provided according to at least one embodiment of the present disclosure;

[0036] FIG14 is a schematic diagram of the main flow of a positioning method provided according to at least one embodiment of the present disclosure;

[0037] FIG15 is a flowchart illustrating steps of obtaining relative positioning of the head mounted device and the neck hanging component based on the first inertial navigation data and image data according to at least one embodiment of the present disclosure; and

[0038] FIG16 is a schematic diagram of the main flow of a sound playing method provided according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0040] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0041] In some examples, VR / AR can adopt an all-in-one solution, where the display, processor, battery, speaker and other devices are all set in the VR / AR glasses. However, this has the problems of large size and weight, and wearing it for a long time will cause users to feel fatigue, discomfort and other symptoms.

[0042] In other examples, a split VR / AR device can be used. This device can be divided into two sub-devices, for example, a head-mounted sub-device (i.e., the glasses portion) and a neck-mounted sub-device, with the larger components, such as the processor, battery, and speakers, placed in the neck-mounted sub-device. The ample space above the neck allows for more, larger, and more widely distributed speakers, resulting in better spatial audio. Furthermore, split VR / AR can effectively alleviate user fatigue compared to all-in-one VR / AR. For solutions where the speakers are mounted on the neck-mounted sub-device, localizing the speakers within the human head coordinate system (hereinafter referred to as the head coordinate system) is essential for implementing spatial audio. Spatial audio requires unifying the human ear, speakers, and virtual sound sources into a single coordinate system. The human ear and virtual sound sources can be located within the glasses coordinate system, which only has an offset, not a rotation, and can therefore be easily converted to the head coordinate system. However, the speakers are not mounted on the glasses, so the speaker coordinates in either the glasses coordinate system or the head coordinate system must be determined. An IMU (Inertial Measurement Unit) or pose estimation sensor can be used to estimate the pose of both sub-devices. The IMU solution is based on integration and is fast but suffers from significant drift. Pose estimation sensors are not affected by line of sight, but are bulky and expensive.

[0043] At least one embodiment of the present disclosure provides a parameter determination method for a split-type device, a parameter determination device for the split-type device, an electronic device, and a computer-readable storage medium. The split-type device includes a first sub-device and a second sub-device in a communicative connection, wherein the second sub-device includes an audio output device and an image acquisition device. The parameter determination method includes: acquiring an image captured by the image acquisition device toward the first sub-device; determining a relative position between the first sub-device and the second sub-device based on the image; and determining operating parameters of the audio output device based on the relative position.

[0044] In this parameter determination method, an audio output device and an image acquisition device are installed on the second sub-device of a split device. The image acquisition device captures images facing the first sub-device. The relative position of the first and second sub-devices is determined based on the images, and the operating parameters of the audio output device are determined based on the relative position. Based on this solution, the relative position of the first and second sub-devices can be accurately and reliably determined through images. The sound information output by each audio output device can then be adjusted in real time based on the relative position, creating a sense of spatial immersion.

[0045] For example, a split device includes a first sub-device and a second sub-device that are communicatively connected, and the second sub-device includes an audio output device and an image acquisition device. In the following embodiments, the wearing part of the first sub-device is the head, the wearing part of the second sub-device is the neck, the audio output device is a speaker, and the image acquisition device is a camera, for example, a grayscale camera or a color camera. However, the present disclosure is not limited to this, and the wearing parts of the first sub-device and the second sub-device can be set as needed, and the second sub-device can also be worn on the waist, shoulder or arm, for example. The split device is a split wearable device, and the split wearable device can be a smart glasses with AR, VR, or MR display functions; it can also be a device without a display function, for example, a head massage device, which in some scenarios also needs to be massaged to make people more relaxed with soothing stereo music.

[0046] FIG1 shows a flow chart of a parameter determination method provided by at least one embodiment of the present disclosure.

[0047] As shown in Figure 1, the parameter determination method may include steps S110 to S130. Step S110: Acquire an image captured by an image capture device of the second sub-device in the direction of the first sub-device.

[0048] Step S120: Based on the image, determine the relative position of the first sub-device and the second sub-device.

[0049] Step S130: Determine operating parameters of the audio output device based on the relative posture.

[0050] For example, in step S110, while the split device is being worn, the image capture device of the second sub-device is oriented toward the first sub-device, and the image captured by the image capture device includes at least a portion of the first sub-device. For example, one or more cameras are provided on both sides of the second sub-device, and the cameras on both sides of the second sub-device are located on both sides of the neck, respectively, for capturing images of both sides of the first sub-device.

[0051] For example, step S120 may include: inputting feature information corresponding to the image into a posture recognition model to determine, based on the posture recognition model, conversion parameters between a first coordinate system and a second coordinate system constructed based on the physical points of the image acquisition device, wherein the conversion parameters are used to represent the relative posture of the first sub-device and the second sub-device, wherein the first coordinate system is constructed based on the first sub-device or the wearing location of the first sub-device. The posture recognition model can be a neural network model or other machine learning regression model, and the posture recognition model can be pre-trained based on sample data. The feature information of the image captured by the image acquisition device is input into the posture recognition model to obtain the output of the posture recognition model, and the output of the posture recognition model can be the conversion parameters between the first coordinate system and the second coordinate system. In some embodiments, the first coordinate system can be constructed based on the physical points of the first sub-device. If the first sub-device is a pair of glasses, the first coordinate system can also be called a glasses coordinate system. The glasses coordinate system can be constructed with a certain point of the glasses (such as the center point of the connecting frame of the two lenses) as the origin, and the glasses coordinate system includes three axes: X / Y / Z. In other embodiments, the first coordinate system may be constructed based on the wearing part of the first sub-device. When the wearing part is the head, the first coordinate system may also be referred to as the head coordinate system. The head coordinate system may be a coordinate system constructed based on a certain point on the head (such as the center point of the line connecting the two ears) as the origin. The head coordinate system includes three axes: X / Y / Z. For example, with the center point of the line connecting the two ears as the origin, the Z axis may be the direction of the face, the Y axis may be toward the top of the head, and the X axis may be toward the ear. The directions of the three axes of the glasses coordinate system may be consistent with the directions of the three axes of the head coordinate system. For example, the second coordinate system is a coordinate system constructed with a certain physical point of the image acquisition device as the origin. If the image acquisition device is a camera, the origin of the second coordinate system may be the optical center of the camera (such as the center of the lens), the Z axis of the second coordinate system may be the direction of the lens observation, the X axis may be toward the bottom or top of the lens, and the X axis may be toward the left or right of the lens. For example, the conversion parameters between two coordinate systems may include a rotation parameter and an offset parameter of one coordinate system relative to the other coordinate system. The rotation parameter may be represented by a quaternion, and the offset parameter may be represented by an offset on the three axes X / Y / Z.

[0052] For example, step S130 may include determining operating parameters of the audio output device based on conversion parameters between the first coordinate system and the second coordinate system. The coordinate values ​​of the audio output device in the first coordinate system may be first determined based on the conversion parameters, and then the operating parameters of the audio output device may be determined based on the coordinate values ​​of the audio output device in the first coordinate system. For example, if the first coordinate system is a glasses coordinate system, the coordinates of the speaker in the glasses coordinate system may be obtained based on the conversion parameters in step S120. Then, based on the offset parameters between the glasses coordinate system and the head coordinate system, the speaker and the virtual sound source point on the glasses may be converted to the head coordinate system. In the head coordinate system, the sound parameters of the speaker may be determined based on the coordinates of the speaker, the virtual sound source point, and the ear. For example, if the first coordinate system is a head coordinate system, the coordinates of the speaker in the head coordinate system may be obtained based on the conversion parameters in step S120. Based on the offset parameters between the glasses coordinate system and the head coordinate system, the virtual sound source point on the glasses may be converted to the head coordinate system. In the head coordinate system, the sound parameters of the speaker may be determined based on the coordinates of the speaker, the virtual sound source point, and the ear. The process of determining sound parameters based on the coordinates of the speaker, the virtual sound source point, and the ear will be described in detail in subsequent embodiments.

[0053] For example, the parameter determination method can be executed by a processor of a split device, and the processor can be set in the second sub-device or in the first sub-device. The first sub-device and the second sub-device can be connected to each other by wired (such as a connecting line) or wireless communication, and the wireless communication can include WIFI, Bluetooth, etc. For example, the image acquisition device is located on the head-mounted sub-device, and the acquired image is transmitted to the processor in the neck-mounted sub-device through wireless technology and other means. The wireless technology can be selected from Bluetooth technology, wireless serial port, and WiFi, or a wireless network communication technology that complies with the 5G WiFi communication protocol can be used to ensure data transmission rate and data throughput.

[0054] According to the parameter determination method of at least one embodiment of the present disclosure, an audio output device and an image acquisition device are provided on the second sub-device of a split device. The image acquisition device captures an image facing the first sub-device, and the relative position between the first and second sub-devices is determined based on the image. The operating parameters of the audio output device are then determined based on the relative position. Based on this solution, the relative position of the first and second sub-devices can be accurately and reliably determined using images. Furthermore, based on the relative position, the sound information output by each audio output device can be adjusted in real time, creating a sense of spatial immersion.

[0055] According to the parameter determination method of at least one embodiment of the present disclosure, image feature information is input into a pose recognition model to obtain conversion parameters between a first coordinate system and a second coordinate system. The coordinates of the audio output device are then converted based on these conversion parameters to determine the operating parameters of the audio output device. This approach allows for accurate and reliable conversion between the first and second coordinate systems with minimal computational effort.

[0056] For example, the first sub-device includes a plurality of markers, and the image includes at least some of the plurality of markers. The parameter determination method may further include: extracting geometric features of at least some of the markers from the image, wherein the geometric features include at least one of a centroid, a slope, and a convex hull area; and wherein the feature information corresponding to the image includes the geometric features.

[0057] FIG2 shows a schematic diagram of a split-type device provided by at least one embodiment of the present disclosure.

[0058] As shown in Figure 2, the split device includes a first sub-device 210 (such as a head-mounted device) and a second sub-device 220 (such as a neck-mounted device). The second sub-device 220 is provided with multiple image acquisition devices 221 and multiple audio output devices 222. The first sub-device is provided with multiple markers 211, and the multiple markers 211 are distributed on both sides of the first sub-device. The image acquisition devices 221 located on both sides of the second sub-device 220 can respectively capture images of the markers 211 on both sides of the first sub-device. For example, the second sub-device includes a first image acquisition device and a second image acquisition device. Three markers are respectively provided on both sides of the first sub-device. The first image acquisition device is used to capture images of the three markers on the left as the first image, and the second image acquisition device is used to capture images of the three markers on the right as the second image. The geometric features such as the center of mass, slope, and convex hull area of ​​the three markers in the first image and the second image are extracted respectively and input into the posture recognition model as feature information corresponding to the image. The centroid can be the centroid of the marker points, the slope can be the slope of the line connecting the marker points, and the convex hull area can be the area of ​​the smallest ellipse or rectangle that can enclose the three marker points. The marker points can be points, rectangles, circles, polygons, or other shapes, and can use different colors, materials, or even reflective materials. Inputting the geometric features of the marker points into the pose recognition model to obtain the conversion parameters can more accurately reflect the pose difference between the first coordinate system and the second coordinate system.

[0059] For example, the first sub-device is provided with a positioning device, which includes: a luminous light source or a reflective light source provided on the first sub-device, used to position the first sub-device when the image acquisition device obtains image data of the first sub-device.

[0060] For example, a luminous light source or a reflective light source can be selected according to the aesthetics of the actual product.

[0061] For example, in one embodiment, the reflective light source is a retroreflective film. Retroreflective film is a marking patch used for stereo positioning. High reflectivity retroreflective film is selected to increase brightness and return incident light along its original path. This makes the retroreflective film very bright, much brighter than the surrounding diffusely reflected light.

[0062] For example, in one embodiment, the light source is an LED lamp, but may also be other types of light sources.

[0063] For example, in one embodiment, the image acquisition device is a monocular camera. When a monocular camera is used, there are at least two corresponding light sources, so as to obtain a stereoscopic image of the first sub-device.

[0064] For example, in one embodiment, the image acquisition device is a binocular camera. Since the binocular camera itself can recognize depth information, the corresponding light source can be one.

[0065] For example, when there are multiple light sources, they should be distributed as dispersedly as possible with a certain interval, which is conducive to image recognition and positioning.

[0066] For example, in one embodiment, when the light source is a reflective light source, the second sub-device further includes an auxiliary light source for illuminating the reflective light source when the image acquisition device is shooting.

[0067] For example, in one embodiment, the auxiliary light source is an infrared light source. Infrared light is invisible light, which makes the product more beautiful and can also be used at night.

[0068] For example, in one embodiment, the image acquisition device includes a near-infrared filter, which is attached to the camera head. When used during the day, the near-infrared light source filter selects to retain light in the 850nm or 940nm band, thereby suppressing background light interference and improving imaging quality.

[0069] For example, near-infrared filters can be selected as long-wave pass filters or narrow-band filters.

[0070] For example, in one embodiment, when the image acquisition device is a monocular camera, the infrared light source is arranged next to the monocular camera; when the image acquisition device is a binocular camera, the infrared light source is arranged between the binocular cameras.

[0071] For example, the FOV of the image acquisition device lens and the luminous angle of the infrared light source can cover the entire head-mounted display device.

[0072] For example, in one embodiment, the first sub-device may not need to be equipped with a light source, and a camera may be used to capture characteristic locations of the first sub-device.

[0073] For example, in one embodiment, a non-luminous marker patch is provided on the first sub-device, and the patch is in a striking color, thus having a positioning function similar to that of a luminous light source.

[0074] For example, in one embodiment, the first sub-device includes a body and a connection piece for connecting to the wearer's ear, wherein the light source is disposed on the connection piece.

[0075] For example, in an application scenario, the first sub-device is a head-mounted display device, the main body is a display glasses part with optical display lenses, and the connecting part is a temple for mounting the display glasses part on the ear.

[0076] For example, the light source is set on the temple, and the relative position of the ear and the temple is known, so the ear can be easily located by locating the temple.

[0077] For example, in some embodiments, the first sub-device may not have markers set, and the image acquisition device may capture an image of the first sub-device itself. In this case, pixel information of all or a portion of the pixels in the image may be input into the pose recognition model as feature information to obtain conversion parameters. For example, the pixel information of the entire image may be input into the pose recognition model, or a portion of the image containing the first sub-device may be extracted from the image, and the pixel information of this portion may be input into the pose recognition model.

[0078] For example, in some embodiments, when the second sub-device includes multiple image acquisition devices, step S120 may include: separately inputting feature information corresponding to images acquired by at least some of the multiple image acquisition devices into the posture recognition model to obtain candidate conversion parameters corresponding to the at least some of the image acquisition devices; and processing the candidate conversion parameters corresponding to the at least some of the image acquisition devices to obtain conversion parameters.

[0079] For example, candidate conversion parameters corresponding to the at least part of the image acquisition devices are processed to obtain conversion parameters, including: determining the measured coordinates of the marking point in the captured image for each image acquisition device in the at least part of the image acquisition devices, determining the reprojection coordinates of the marking point based on the corresponding candidate conversion parameters, and taking the deviation between the measured coordinates and the reprojection coordinates as the error of the candidate conversion parameter; and selecting a candidate conversion parameter with the smallest error among the candidate conversion parameters corresponding to the at least part of the image acquisition devices as the conversion parameter.

[0080] FIG3 shows a schematic diagram of an image acquisition and processing process provided by at least one embodiment of the present disclosure.

[0081] As shown in Figure 3, the second sub-device 320 includes a first image acquisition device and a second image acquisition device (both denoted by reference numeral 321), as well as multiple audio output devices 322. The first sub-device 310 includes multiple markers located on either side of the device. Feature information (e.g., geometric features of the markers) of the image captured by the first image acquisition device is input into a pose recognition model to obtain first candidate transformation parameters (pose 1) corresponding to the first image acquisition device. Feature information (e.g., geometric features of the markers) of the image captured by the second image acquisition device is input into the pose recognition model to obtain second candidate transformation parameters (pose 2) corresponding to the second image acquisition device. The pose estimation reprojection error corresponding to the first candidate transformation parameter and the pose estimation reprojection error corresponding to the second candidate transformation parameter are calculated, and the one with the smaller error is selected as the transformation parameter between the first coordinate system and the second coordinate system. For example, taking the first camera among the multiple cameras as an example, the image captured by the first camera includes markers 1 and 2. The image captured by the first camera is input into the pose recognition model to obtain candidate transformation parameters corresponding to the first camera. Extract the measured coordinates of marker points 1 and 2 from the image. Based on the coordinates of marker points 1 and 2 in the first coordinate system and the candidate transformation parameters, obtain the reprojected coordinates of marker points 1 and 2. The deviation between the measured and reprojected coordinates of marker points 1 and 2 is used as the error in the candidate transformation parameters for the first camera. The same process is repeated for other cameras. From the multiple candidate transformation parameters corresponding to the multiple cameras, the one with the smallest error is selected as the transformation parameter between the first coordinate system and the second coordinate system.

[0082] For example, in some other embodiments, after obtaining candidate conversion parameters corresponding to multiple cameras respectively, the conversion parameters between the first coordinate system and the second coordinate system may be obtained by performing weighted averaging on the multiple candidate conversion parameters.

[0083] For example, in other embodiments, when the second sub-device includes multiple image acquisition devices, step S120 may include: inputting features corresponding to images acquired by multiple image acquisition devices into the posture recognition model together to obtain conversion parameters corresponding to the multiple image acquisition devices.

[0084] For example, feature information of images captured by the first image acquisition device and feature information of images captured by the second image acquisition device can be input into a pose recognition model to obtain conversion parameters between the first coordinate system and the second coordinate system. For example, by inputting feature information of images from multiple cameras into the pose recognition model, the pose recognition model can be used to fuse the images from the multiple cameras. The conversion parameters output by the model are the result of fusing the images from the multiple cameras. Therefore, the conversion parameters obtained in this manner are more accurate.

[0085] For example, as described above, in step S130, the coordinate values ​​of the audio output device in the first coordinate system may be first determined based on the conversion parameters, and then the operating parameters of the audio output device may be determined based on the coordinate values ​​of the audio output device in the first coordinate system. In some embodiments, determining the coordinate values ​​of the audio output device in the first coordinate system based on the conversion parameters may include determining the coordinate values ​​of the audio output device in the first coordinate system based on the conversion parameters and the coordinates of the audio output device in the second coordinate system.

[0086] For example, the conversion parameters between the first coordinate system and the second coordinate system may be updated at predetermined intervals (e.g., 0.5 seconds or 1 second). After each update, the current coordinates of the speaker in the first coordinate system are calculated based on the coordinates of the speaker in the second coordinate system and the updated conversion parameters. The frequency of updating the conversion parameters may be determined based on the frequency of image acquisition by the image acquisition specialist.

[0087] For example, in other embodiments, determining the coordinate value of the audio output device in the first coordinate system based on the conversion parameters may include: determining the coordinate measurement value of the audio output device in the first coordinate system at the first moment based on the conversion parameters determined at the first moment and the coordinate of the audio output device in the second coordinate system; determining the coordinate prediction value of the audio output device in the first coordinate system at the first moment based on historical coordinate measurement values ​​obtained within a period of time before the first moment, wherein the historical coordinate measurement value is the coordinate measurement value of the audio output device in the first coordinate system obtained based on images obtained within a period of time before the first moment; and fusing the coordinate measurement value with the coordinate prediction value to obtain a fused coordinate value, and using the fused coordinate value as the coordinate value of the audio output device in the first coordinate system at the first moment.

[0088] FIG4 shows a flowchart of another parameter determination method provided by at least one embodiment of the present disclosure.

[0089] As shown in Figure 4, the first coordinate system is, for example, the eyeglass coordinate system, and the second coordinate system is, for example, the camera coordinate system. In step S401, an image is captured using the camera on the second sub-device as the image at a first moment (e.g., the current moment). In step S402, geometric features such as the centroid, slope, and convex hull area of ​​the markers are extracted from the image. In step S403, the extracted features are input into a pose recognition model to obtain pose estimates for the camera coordinate system and the eyeglass coordinate system, that is, to obtain conversion parameters between the two coordinate systems, which serve as conversion parameters at the first moment. In step S404, the coordinates of the speaker in the camera coordinate system are obtained. In step S405, the coordinate measurement values ​​of the speaker in the eyeglass coordinate system at the first moment are calculated based on the coordinates of the speaker in the camera coordinate system and the conversion parameters at the first moment. In step S408, a Kalman filter is used to obtain the predicted coordinate values ​​of the speaker in the eyeglass coordinate system at the first moment. In step S409, the measured coordinate values ​​and the predicted coordinate values ​​at the first moment are fused to obtain fused coordinate values. The Kalman filter is an algorithm that continuously corrects the current state based on historical states. This algorithm is divided into measurement values, prediction values, and fusion values. The prediction value predicts the current state value based on historical states. The measurement value is the state value actually measured by the sensor, and the fusion value is the output value that combines the current state measurement value with the prediction value. When the measurement value is affected by noise, anomalies, or other interference, the fusion value does not directly depend on the measurement value. Instead, it makes appropriate corrections based on the prediction value of the historical state. Even when a sensor error occurs within a short period of time, it can output a more reliable value based on its historical state. This algorithm can significantly improve the measurement stability of the entire system in scenarios with high measurement noise, thereby enhancing the system's measurement accuracy and robustness.

[0090] For example, when the side of the first sub-device facing the second sub-device is blocked, the coordinate measurement value obtained based on the image may be inaccurate. In this case, the coordinate prediction value is obtained using historical data, and then the coordinate prediction value and the coordinate measurement value are fused to obtain a coordinate fusion value, which can more accurately reflect the position of the audio output device in the first coordinate system. During the fusion process of the coordinate prediction value and the coordinate measurement value, it can be determined whether the coordinate measurement value is within the expected range. Since the posture of the device usually changes linearly, the expected range can be determined based on the historical coordinate data. If the coordinate measurement value is within the expected range, the coordinate measurement value and the coordinate prediction value can be weighted averaged to obtain the coordinate fusion value.

[0091] For example, before step S408, steps S406 to S407 may also be included. In step S406, the parameters of the Kalman filter are set. In step S407, the parameters of the Kalman filter may be updated based on the historical coordinate data before the first moment to obtain a Kalman filter with updated parameters. The historical coordinate data before the first moment may be coordinate measurement values ​​obtained within a period of time before the first moment, and the parameters of the Kalman filter may be updated once every predetermined time period (e.g., 0.5 seconds or 1 second). Then, in step S408, the Kalman filter with updated parameters may be used to determine the coordinate prediction value of the audio output device in the first coordinate system at the first moment.

[0092] For example, in other embodiments, when it is detected at the first moment that the first sub-device is blocked, the coordinate prediction value of the audio output device at the first moment in the first coordinate system can be obtained according to the Kalman filter, and the coordinate prediction value at the first moment can be used as the coordinate value of the audio output device at the first moment in the first coordinate system.

[0093] For example, in some embodiments, determining operating parameters of the audio output device based on the coordinate values ​​of the audio output device in a first coordinate system may include: determining the coordinate values ​​of a virtual sound source point on the first sub-device in the first coordinate system based on conversion parameters between the first coordinate system and a third coordinate system constructed with a physical point of the first sub-device, when the first coordinate system is constructed based on the wearing location of the first sub-device; and determining the operating parameters of the audio output device in the first coordinate system based on the coordinate values ​​of the virtual sound source point and the coordinate values ​​of the audio output device. For example, in this example, the first coordinate system may be a head coordinate system constructed with a certain point on the head as the origin, and the third coordinate system may be a coordinate system constructed with the center point of the glasses (e.g., the center point of the connecting frame of the two lenses) as the origin.

[0094] For example, in other embodiments, determining operating parameters of the audio output device based on the coordinate values ​​of the audio output device in a first coordinate system may include: when the first coordinate system is constructed based on the first sub-device, determining the coordinate values ​​of a virtual sound source point on the first sub-device in the fourth coordinate system and the coordinate values ​​of the audio output device in the fourth coordinate system based on conversion parameters between the first coordinate system and a fourth coordinate system (e.g., a head coordinate system) constructed based on the wearing location of the first sub-device (e.g., the head); and determining the operating parameters of the audio output device in the fourth coordinate system based on the coordinate values ​​of the virtual sound source point and the coordinate values ​​of the audio output device. For example, in this example, the first coordinate system may be a coordinate system constructed with the center point of the glasses (e.g., the center point of the connecting frame of the two lenses) as the origin, and the fourth coordinate system may be a head coordinate system constructed with a certain point on the head (e.g., the center point of the line connecting the two ears) as the origin.

[0095] For example, speakers can be used to simulate sounds at virtual sound source points to achieve spatial audio.

[0096] In the embodiment according to the present disclosure, four coordinate systems are defined: a first, second, third, and fourth coordinate systems. These coordinate systems are constructed based on specific reference points and are used to accurately capture and define the position and posture of each part of the split device. By clarifying the conversion relationship between each coordinate system, precise control of the movement and positioning of each part of the device can be achieved. One of the above coordinate systems, such as the first coordinate system, can be used as a world coordinate system, but this embodiment is not limited to this. The world coordinate system can be freely set according to the computing requirements to adapt to different control strategies.

[0097] FIG5 shows a schematic diagram of determining sound parameters provided by at least one embodiment of the present disclosure.

[0098] As shown in Figure 5, after unifying the human ear, speaker, and virtual sound source point into the head coordinate system, the coordinate of the speaker is P A , the position of the virtual sound source is P B , according to the coordinates of the virtual sound source and the sound parameters S of the virtual sound source B , and the coordinates of the human ear, calculate the sound E of the virtual sound source heard by the human ear B If the sound from the speaker is S A , then the sound of the speaker heard by the human ear is E A The speaker needs to simulate the sound of the virtual sound source, so E A =E B , based on this, the sound parameters of the speaker can be calculated.

[0099] At least one embodiment of the present disclosure provides a split device, which includes a first sub-device and a second sub-device, and the second sub-device includes an audio output device and an image acquisition device; wherein the first sub-device or the second sub-device includes a processor, the image acquisition device is used to capture images in the direction of the first sub-device and send the captured images to the processor, the processor is used to determine the relative position of the first sub-device and the second sub-device based on the image, determine the operating parameters of the audio output device based on the relative position, and send the operating parameters to the audio output device.

[0100] FIG6 shows a schematic block diagram of a parameter determination apparatus 600 provided by at least one embodiment of the present disclosure.

[0101] For example, as shown in FIG6 , the parameter determination device 600 includes an acquisition unit 610, a conversion unit 620, and a determination unit 630. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, these modules can be implemented by hardware (e.g., circuit) modules, software modules, or any combination thereof, and the following embodiments are the same and will not be described in detail. For example, these units can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, as well as corresponding computer instructions. It should be noted that the components and structures of the parameter determination device 600 shown in FIG6 are exemplary only and not restrictive. As needed, the parameter determination device 600 may also have other components and structures.

[0102] The acquisition unit 610 is configured to acquire an image acquired by the image acquisition device in the direction of the first sub-device. The acquisition unit 610 may, for example, execute step S110 described in FIG1 .

[0103] The conversion unit 620 is configured to determine the relative position of the first sub-device and the second sub-device based on the image. The conversion unit 620 can, for example, execute step S120 described in FIG1 .

[0104] The determining unit 630 is configured to determine the operating parameters of the audio output device based on the conversion parameters. The determining unit 630 may, for example, execute step S130 described in FIG1 .

[0105] For example, the acquisition unit 610, the conversion unit 620, and the determination unit 630 may be hardware, software, firmware, or any feasible combination thereof. For example, the acquisition unit 610, the conversion unit 620, and the determination unit 630 may be dedicated or general-purpose circuits, chips, or devices, or may be a combination of a processor and memory. The embodiments of the present disclosure do not limit the specific implementation of each of the above units.

[0106] For example, the acquisition unit 610, the conversion unit 620, and the determination unit 630 may include codes and programs stored in a memory; the processor may execute the codes and programs to implement some or all of the functions of the image acquisition unit 610, the conversion unit 620, and the determination unit 630 as described above. For example, the acquisition unit 610, the conversion unit 620, and the determination unit 630 may be dedicated hardware devices used to implement some or all of the functions of the acquisition unit 610, the conversion unit 620, and the determination unit 630 as described above. For example, the acquisition unit 610, the conversion unit 620, and the determination unit 630 may be a circuit board or a combination of multiple circuit boards used to implement the functions described above. In an embodiment of the present disclosure, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processors; and (3) firmware stored in the memory that is executable by the processor.

[0107] It should be noted that, in the embodiment of the present disclosure, the various units of the parameter determination device 600 correspond to the various steps of the aforementioned parameter determination method. For the specific functions of the parameter determination device 600, reference can be made to the relevant description of the parameter determination method, which will not be repeated here. The components and structure of the parameter determination device 600 shown in Figure 6 are only exemplary and not restrictive. As needed, the parameter determination device 600 may also include other components and structures. The parameter determination device 600 may include more or fewer circuits or units, and the connection relationship between each circuit or unit is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited, and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or can be composed in other applicable ways.

[0108] At least one embodiment of the present disclosure further provides an electronic device including a processor and a memory, wherein the memory stores one or more computer program modules configured to be executed by the processor to implement the above-mentioned parameter determination method.

[0109] Figure 7 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure. As shown in Figure 7, the electronic device 700 includes a processor 710 and a memory 720. The memory 720 stores non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 710 is used to run non-transitory computer-readable instructions, and the non-transitory computer-readable instructions are executed by the processor 710 when they are executed. One or more steps in the parameter determination method described above are executed. The memory 720 and the processor 710 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown). For the specific implementation of each step of the parameter determination method and the related explanation content, please refer to the embodiment of the parameter determination method mentioned above, and the repetitions will not be repeated here.

[0110] It should be noted that the components of the electronic device 700 shown in FIG. 7 are merely exemplary and non-limiting. The electronic device 700 may further include other components according to actual application requirements.

[0111] For example, the processor 710 and the memory 720 may communicate with each other directly or indirectly.

[0112] For example, the processor 710 and the memory 720 may communicate via a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 710 and the memory 720 may also communicate with each other via a system bus, which is not limited in this disclosure.

[0113] For example, the processor 710 and the memory 720 may be provided on a server side (or a cloud side).

[0114] For example, the processor 710 can control other components in the electronic device 700 to perform desired functions. For example, the processor 710 can be a central processing unit (CPU), a graphics processing unit (GPU), or other processing units with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) can be an X86 or ARM architecture. The processor 710 can be a general-purpose processor or a dedicated processor, and can control other components in the electronic device 700 to perform desired functions.

[0115] For example, the memory 720 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 710 may execute one or more computer program modules to implement various functions of the electronic device 700. Various applications and various data, as well as various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0116] It should be noted that, in the embodiment of the present disclosure, the specific functions and technical effects of the electronic device 700 can be referred to the description of the parameter determination method above, which will not be repeated here.

[0117] Figure 8 is a schematic block diagram of another electronic device provided in some embodiments of the present disclosure. This electronic device 800 is, for example, suitable for implementing the parameter determination method provided in embodiments of the present disclosure. The electronic device 800 may be a terminal device, etc. It should be noted that the electronic device 800 shown in Figure 8 is merely an example and does not impose any limitations on the functionality and scope of use of the embodiments of the present disclosure.

[0118] As shown in FIG8 , the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 810, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 820 or a program loaded from a storage device 880 into a random access memory (RAM) 830. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 830. The processing device 810, the ROM 820, and the RAM 830 are connected to each other via a bus 840. An input / output (I / O) interface 850 is also connected to the bus 840.

[0119] Typically, the following devices may be connected to the I / O interface 850: an input device 860 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 870 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 880 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 890. The communication device 890 may allow the electronic device 800 to communicate with other electronic devices wirelessly or by wire to exchange data. Although FIG8 shows the electronic device 800 with various devices, it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 800 may alternatively implement or have more or fewer devices.

[0120] For example, according to an embodiment of the present disclosure, the above-mentioned parameter determination method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the above-mentioned parameter determination method. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 890, or installed from the storage device 880, or installed from the ROM 820. When the computer program is executed by the processing device 810, the functions defined in the parameter determination method provided in the embodiment of the present disclosure can be implemented.

[0121] At least one embodiment of the present disclosure further provides a computer-readable storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the above-mentioned parameter determination method can be implemented.

[0122] Figure 9 is a schematic diagram of a storage medium provided by some embodiments of the present disclosure. As shown in Figure 9, storage medium 900 stores non-transitory computer-readable instructions 910. For example, when non-transitory computer-readable instructions 910 are executed by a computer, one or more steps of the parameter determination method described above are performed.

[0123] For example, the storage medium 900 can be applied to the electronic device 700 described above. For example, the storage medium 900 can be the memory 720 in the electronic device 700 shown in FIG7 . For example, the relevant description of the storage medium 900 can refer to the corresponding description of the memory 720 in the electronic device 700 shown in FIG7 , and will not be repeated here.

[0124] At least one embodiment of the present disclosure provides a positioning device for a split-type device. Figure 10 shows a schematic block diagram of a positioning device for a split-type device provided by at least one embodiment of the present disclosure. As shown in Figure 10, the split-type wearable device includes a first sub-device 1 and a second sub-device 2, wherein the second sub-device is provided with an audio output device 3. The positioning device includes: a first inertial measurement unit 4 provided on the first sub-device, for obtaining first inertial navigation data of the first sub-device; an image acquisition device 5 provided on the second sub-device, for obtaining image data of the first sub-device; and a processor for determining the relative positioning of the first sub-device and the second sub-device based on the first inertial navigation data and the image data.

[0125] For example, in the following embodiments, the first sub-device may be a head-mounted device, the second sub-device may be a neck pendant, the audio output device may be a speaker, the image acquisition device may also be called an image acquisition unit, and the processor may also be called a controller.

[0126] For example, in one embodiment, the first inertial measurement unit is an IMU (Inertial Measurement Unit), which is a sensor mainly used to detect and measure acceleration and rotational motion.

[0127] For example, an IMU primarily measures two physical quantities: angular velocity, which can be integrated to determine the object's rotation angle, and acceleration, which can be integrated to determine the object's speed and distance. These two quantities provide an object's short-term trajectory. When in use, the IMU moves with the headset, collecting real-time information about the headset's position, including translation and rotation.

[0128] For example, the first inertial navigation data includes angular velocity and acceleration data of the head-mounted device. The first inertial navigation data also includes posture information of the head-mounted device, and the posture information is obtained based on the angular velocity and acceleration data.

[0129] For example, the image acquisition unit may be a camera, and the image data of the head-mounted device includes images captured by a camera head. The camera captures images of the current viewing angle through the camera head. The camera head is mounted on a neck pendant and faces the head-mounted device to capture images of the head-mounted device relative to the camera head.

[0130] For example, the advantage of IMU is that it can provide the relative motion displacement of the head-mounted device. The disadvantage is the error accumulation and drift of IMU. Since IMU is only applicable to posture estimation in a short period of time, the error accumulation caused by drift of IMU makes it difficult for IMU to achieve long-term direction estimation. Therefore, the image acquired by the image acquisition unit of the present invention is used to obtain the absolute position information of the head-mounted device to correct the relative displacement of IMU. The correction method can use a method that combines inertial positioning and visual positioning in the prior art, such as a filter-based method or an optimization / bundle adjustment (BA)-based method to achieve correction, and obtain an accurate relative motion displacement of the head-mounted device after correction.

[0131] For example, the camera captures images of the head-mounted device at intervals to obtain the position relationship of the head-mounted device relative to the camera. In the interval between two image captures, the IMU is used to obtain the relative position of the head-mounted device. As a supplement to image capture, the IMU is used to capture the posture of the head-mounted device between two adjacent photographic images to obtain the translation and rotation trajectories of the head-mounted device, thereby realizing the positioning of the head-mounted device within the interval between photographs. Although the IMU has drift, after the above corrections, accurate IMU posture capture data of the head-mounted device is obtained.

[0132] For example, by combining images and IMU, the relative positioning of the head-mounted device and the neck pendant can be obtained.

[0133] For example, in an application scenario, the positional relationship of the head-mounted device relative to the camera head is obtained through the image captured by the camera. With the camera head as the origin, the three-dimensional coordinates of the head-mounted device in the camera coordinate system can be obtained. The installation position of the camera on the neck pendant is known, and the three-dimensional coordinates of the neck pendant in the camera coordinate system can be obtained. The positional relationship between the head-mounted device and the neck pendant is obtained through the camera coordinate system to realize the positioning of the head-mounted device and the neck pendant on the image.

[0134] For example, in one embodiment, the neck hanging part further includes: speakers, including at least two, whose positions correspond to the two ears of the wearer of the head-mounted device, wherein the controller obtains the relative positioning of the speakers and the ears based on the relative positioning of the head-mounted device and the neck hanging part.

[0135] For example, the positions corresponding to the two ears of the wearer of the head-mounted device are the projection range of the ears on the neck pendant in the front-facing state. The head-mounted device is mounted on the wearer's head through the wearer's ears. The position of the wearer's ears relative to the head-mounted device in the front-facing state is known, and the installation position of the speaker on the neck pendant is known. For example, two speakers can be installed, and when the wearer looks straight ahead, there is one speaker under each ear. Or four speakers can be installed, and when the wearer looks straight ahead, there are two speakers under each ear. In this way, the relative positioning of the speaker and the ear can be obtained through the relative positioning of the head-mounted device and the neck pendant.

[0136] For example, those skilled in the art will understand that since the position between the wearer's ears and the head-mounted device is relatively fixed, and the positions of the neck pendant and the speaker are relatively fixed, after obtaining the relative positioning of the head-mounted device and the neck pendant, the relative positioning of the ears and the speaker will naturally be obtained.

[0137] For example, the positional relationship between the head-mounted device and the neck pendant constructed with the camera as the origin mentioned in the application scenario described above can be obtained by obtaining the relative positions of the speaker and the ear through methods such as the three-dimensional coordinate system conversion matrix.

[0138] For example, in the above embodiment, as shown in FIG10 , only when an IMU is provided on the head-mounted device, the image acquisition unit needs to have a sufficiently high image acquisition frequency so that the controller can determine the relative positioning of the head-mounted device and the neck pendant based on the first inertial navigation data and the image data. For example, the image acquisition unit uses a high frame rate camera, such as a 60 fps, 90 fps, or higher frame rate camera.

[0139] For example, the image acquisition unit captures images using a visual positioning method. This method utilizes a visual system to capture environmental images during the movement of the mobile device through an imaging device and extracts feature points from each image. The motion of the mobile device is estimated based on changes in these feature points. Inertial positioning uses a first inertial measurement unit to collect first inertial navigation data. Inertial positioning uses a known initial position to infer the next position and attitude based on continuously measured acceleration and angular velocity of the mobile device, thereby estimating the current attitude of the mobile device in real time. While visual positioning can achieve high-precision positioning results, its positioning frequency is too low and there is a risk of positioning failure. Inertial elements offer high short-term accuracy and are subject to precision constraints. However, due to their high output frequency, the accumulated calculated attitude can introduce corresponding cumulative errors, resulting in "drift."

[0140] For example, in theory, if the camera has an infinitely high frame rate, the relative positioning of the head-mounted device and the neck pendant can be completed with only the camera. However, the actual high frame rate brings high energy consumption, so the camera cannot achieve an infinitely high frame rate. In this embodiment, a camera and a single IMU are used together. The camera positions the camera head and the head-mounted device, and the single IMU only positions the head-mounted device. Therefore, the camera still needs to maintain a high frame rate and a high image acquisition frequency to ensure the relative positioning accuracy of the head-mounted device and the neck pendant.

[0141] For example, the present disclosure adopts a vision and inertial fusion method. Since a high frame rate camera is used, the visual positioning result is mainly used. When the visual positioning result appears, the current visual positioning result is output as the positioning result; thereafter, until a new visual positioning result appears, the positioning result is predicted by the posture inferred from the visual positioning result and the inertial element and output as the positioning result; when the next visual positioning result appears, the visual positioning result at the current moment is output as the positioning result.

[0142] For example, an embodiment that does not require a very high acquisition frequency of the camera is given below. To this end, referring to FIG11 , the positioning device further includes: a second inertial measurement unit 6 provided on the neck pendant, for acquiring second inertial navigation data of the neck pendant.

[0143] For example, the controller obtains the relative positioning of the head-mounted device and the neck hanging component based on the first inertial navigation data, the second inertial navigation data, and the image data.

[0144] For example, the second inertial measurement unit is an IMU (Inertial Measurement Unit), which is a sensor mainly used to detect and measure acceleration and rotational motion.

[0145] For example, when in use, the IMU moves along with the movement of the neck pendant, collecting real-time position information such as translation and rotation of the neck pendant.

[0146] For example, the second inertial navigation data includes angular velocity and acceleration data of the neck pendant. The second inertial navigation data also includes posture information of the neck pendant, and the posture information is obtained according to the angular velocity and acceleration data.

[0147] For example, this embodiment adopts the method of cooperating with dual IMU and camera, and the correction method refers to the method of cooperating with single IMU and camera.

[0148] For example, this embodiment uses a camera and dual IMUs. The camera locates the camera head and head-mounted device, one IMU locates the head-mounted device, and the other IMU locates the neck pendant. This configuration can reduce the frequency of camera image acquisition and no longer rely on visual positioning. In the interval between two adjacent image acquisitions, the motion trajectory of the head-mounted device in the interval is obtained by the first inertial measurement unit, and the motion trajectory of the neck pendant in the interval is obtained by the second inertial measurement unit. The drift of the two inertial measurement units is corrected using the image, and finally the position of the head-mounted device relative to the camera is accurately and in real time obtained through the image and the two inertial measurement units, and then the translation and rotation of the head-mounted device relative to the neck pendant are obtained, thereby realizing the relative positioning of the two.

[0149] For example, the method of this embodiment can improve the accuracy and real-time performance of positioning, while greatly reducing the energy consumption of the camera.

[0150] For example, whether in the embodiment shown in Figure 10 or Figure 11, the positioning device may also include: a luminous light source or a reflective light source arranged on the head-mounted device, used to position the head-mounted device when the image acquisition unit acquires image data of the head-mounted device.

[0151] For example, a luminous light source or a reflective light source can be selected according to the aesthetics of the actual product.

[0152] For example, in one embodiment, the reflective light source is a retroreflective film. Retroreflective film is a marking patch used for stereo positioning. High reflectivity retroreflective film is selected to increase brightness and return incident light along its original path. This makes the retroreflective film very bright, much brighter than the surrounding diffusely reflected light.

[0153] For example, in one embodiment, the light source is an LED lamp, or other types of light sources.

[0154] For example, in one embodiment, the image acquisition unit is a monocular camera. When a monocular camera is used, there are at least two corresponding light sources, so as to obtain a stereoscopic image of the head-mounted device.

[0155] For example, in one embodiment, the image acquisition unit is a binocular camera. Since the binocular camera itself can recognize depth information, the corresponding light source can be one.

[0156] For example, when there are multiple light sources, they should be distributed as dispersedly as possible with a certain interval, which is conducive to image recognition and positioning.

[0157] For example, in one embodiment, when the light source is a reflective light source, the neck pendant further includes an auxiliary light source for illuminating the reflective light source when the image acquisition unit is shooting.

[0158] For example, in one embodiment, the auxiliary light source is an infrared light source. Infrared light is invisible light, which makes the product more beautiful and can also be used at night.

[0159] For example, in one embodiment, the image acquisition unit includes a near-infrared filter, which is attached to the camera head. When used during the day, the near-infrared light source filter selects to retain light in the 850nm or 940nm band, thereby suppressing background light interference and improving imaging quality.

[0160] For example, near-infrared filters can be selected as long-wave pass filters or narrow-band filters.

[0161] For example, in one embodiment, when the image acquisition unit is a monocular camera, the infrared light source is arranged next to the monocular camera; when the image acquisition unit is a binocular camera 7, the infrared light source 8 is arranged between the binocular cameras 7, see Figure 12.

[0162] For example, the FOV of the image acquisition unit lens and the luminous angle of the infrared light source can cover the entire head-mounted display device.

[0163] For example, in one embodiment, the head-mounted device may not need to be equipped with a light source, and the camera may capture characteristic locations of the head-mounted device.

[0164] For example, in one embodiment, a non-luminous marker patch is provided on the head-mounted device, and the patch is in a striking color, thus having a positioning function similar to a luminous light source.

[0165] For example, in one embodiment, the head-mounted device includes a body and a connection piece for connecting to the wearer's ear, wherein the light source is disposed on the connection piece.

[0166] For example, in an application scenario, the head-mounted device is a head-mounted display device, the main body is a display glasses part with optical display lenses, and the connecting part is a temple for mounting the display glasses part on the ear.

[0167] For example, the light source is set on the temple, and the relative position of the ear and the temple is known, so the ear can be easily located by locating the temple.

[0168] For example, based on the above positioning device, the present disclosure also provides a split wearable device, referring to FIG13 , which includes: a head-mounted device, a neck pendant, and the positioning device of the above embodiment.

[0169] For example, in one embodiment, the present disclosure further provides a positioning method for a split-type wearable device, referring to FIG14 , including:

[0170] S1410: Acquire first inertial navigation data of the head mounted device through a first inertial measurement unit provided on the head mounted device;

[0171] S1420: Acquire image data of the head mounted device through an image acquisition unit provided on the neck pendant;

[0172] S1430: Obtain, by a controller, the relative positioning of the head-mounted device and the neck pendant based on the first inertial navigation data and the image data.

[0173] For example, in one embodiment, in S1430, obtaining the relative positioning of the head mounted device and the neck hanging component based on the first inertial navigation data and the image data, as shown in FIG. 15 , includes:

[0174] S1431, obtaining absolute position information of the head mounted device according to the image data;

[0175] S1432: Obtain relative position information of the head mounted device according to the first inertial navigation data;

[0176] S1433: Use the absolute position information to correct the relative position information to obtain the relative positioning of the head-mounted device and the neck pendant.

[0177] Next, a specific wearable device is used to describe the process of obtaining the absolute position information of the head-mounted device based on the image data.

[0178] The wearable device's head-mounted device is a display pair of glasses, marked with two retroreflective films spaced apart on the temples. The image acquisition unit is a monocular camera, and an infrared light source is mounted on a neck pendant, positioned near the monocular camera. The retroreflective film is used to retroreflect the incident infrared light.

[0179] The process of obtaining the absolute position information of the head mounted device according to the image data includes:

[0180] (1) an image acquisition device acquires an image;

[0181] (2) Using the threshold segmentation method to segment the inverse film image;

[0182] (3) Determine whether the number of reverse films is sufficient;

[0183] The reflective film may be blocked by human hands. For example, during image acquisition, the wearer may hold the glasses with their hands, resulting in one or more reflective films being blocked. For example, when a monocular camera is used with a reflective film, at least two reflective films must be present when acquiring an image. If there is any blockage, such as only one film being recognized, it is determined that the number of reflective films is insufficient.

[0184] (4) If sufficient, the center of the anti-reflective film is extracted, and the position of the anti-reflective film (such as the temple of the head display device) and the positioning of the camera are calculated using the PNP algorithm or neural network.

[0185] If the PNP algorithm is used, the monocular camera needs at least three anti-reflection films, and if the neural network algorithm is used, at least two anti-reflection films are required.

[0186] Taking the PNP algorithm as an example, camera localization primarily involves estimating the camera's extrinsic parameters [RT]. This is done based on feature points. Using markers as feature points, feature-point-based camera localization is also known as the PNP (Perspective-N-Point) problem, also known as the N-point perspective problem. The PNP problem involves estimating the position and orientation of an observed object relative to the camera from N image points under perspective projection, given a calibrated camera. In abstract terms, given n markers, the distance between each pair of markers and the angle between the line connecting the marker and its image point and the camera's optical center must be calculated. Generally, visual localization based on the PNP problem has multiple solutions. However, for any three points in a plane that are not aligned, visual localization based on the PNP problem has a unique solution. The PNP problem effectively transforms the object localization problem into a mathematical equation.

[0187] (5) The position of the head-mounted display (HMD) is determined based on the location of the marker's reverse film (e.g., the temples of the HMD), and the position of the neck-mounted display is determined based on the location of the camera. (The locations of the temples and HMD are known, and the location of the camera on the neck-mounted display is known.) This allows the HMD and neck-mounted display to be positioned.

[0188] Based on the above positioning method, the present disclosure also provides a sound playing method for a split wearable device, referring to FIG16 , comprising:

[0189] S1610: Obtain relative positioning of the head-mounted device and the neck pendant according to the positioning method for a split-type wearable device;

[0190] S1620: Obtaining relative positions of a speaker provided on the neck hanging piece and an ear of a wearer of the head mounted device based on the relative positions of the head mounted device and the neck hanging piece;

[0191] S1630: Acquire a head-related transfer function corresponding to the speaker based on the relative positioning of the speaker and the ear;

[0192] S1640: Control the speaker to produce sound based on the head-related transfer function.

[0193] In one application scenario, the neck pendant is equipped with four speakers. When the wearer looks straight ahead, two speakers are located under each ear, and the speakers are used to complete the audio playback function. Through relative positioning, an algorithm adjusts the control signal input to the speaker. The speaker receives the control signal and adjusts the playback of each speaker. The specific process is as follows:

[0194] An HRTF library is pre-established. HRTF stands for head-related transfer function. The HRTF library includes the HRTF of each speaker on the neck pendant.

[0195] Based on the relative positioning as input information, the far-field HRTF and near-field HRTF are found from the preset HRTF library by table lookup, and the actual sound emitted by each speaker is adjusted to: (positioning distance * far-field HRTF) / near-field HRTF.

[0196] When the wearer puts on the head-mounted device, the posture changes, for example, from looking straight ahead to turning the head to look straight to the side. At this time, the values ​​of the far-field HRTF and near-field HRTF change, and the positioning distance also changes. According to the above algorithm, the actual sound produced by the speaker is adjusted.

[0197] Those skilled in the art will appreciate that the present disclosure is not limited to the above-mentioned method for controlling the sound emission of a speaker.

[0198] Preferably, in order to prevent the sounds of the left and right ears from crosstalking with each other, a crosstalk cancellation technology may be added, and the specific implementation method is not limited.

[0199] The present invention provides an inertial measurement unit on the head-mounted device or on both the head-mounted device and the neck pendant, and combines it with an image acquisition unit provided on the neck pendant to provide high-speed relative positioning with the inertial measurement unit data, and correct the inertial measurement data with the visual positioning result, so as to be able to locate the relative position between the head-mounted device and the neck pendant in real time, accurately and efficiently.

[0200] The positioning device disclosed herein can be used to obtain the relative position of the ear and the speaker based on the relative position of the head-mounted device and the ear and the installation position of the speaker on the neck pendant.

[0201] The relative positions of the ears and the speakers obtained by the positioning device disclosed in the present invention are used to adjust the sound signals output by each speaker in real time to create a sense of spatial immersion.

[0202] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0203] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0204] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0205] Regarding this disclosure, the following points need to be explained:

[0206] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0207] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0208] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A method for determining parameters of a split device, wherein, The split device includes a first sub-device and a second sub-device that are communicatively connected. The second sub-device includes an audio output device and an image acquisition device. The method includes: Obtain an image acquired by the image acquisition device in the direction of the first sub-device; Based on the image, determine the relative pose of the first sub-device and the second sub-device; Based on the relative pose, determine the operating parameters of the audio output device.

2. The parameter determination method according to claim 1, wherein, Determining the relative pose of the first sub-device and the second sub-device based on the image includes: Input the feature information corresponding to the image into a pose recognition model to determine the transformation parameters between a first coordinate system and a second coordinate system constructed based on physical points of the image acquisition device, where the transformation parameters are used to represent the relative pose of the first sub-device and the second sub-device. Among them, the first coordinate system is constructed based on the first sub-device or the wearing part of the first sub-device.

3. The parameter determination method according to claim 2, wherein The first sub-device includes a plurality of marker points, and the image includes at least some of the plurality of marker points. The method further includes: extracting the geometric features of the at least some marker points from the image, where the geometric features include at least one of a centroid, a slope, and a convex hull area; Among them, the feature information corresponding to the image includes the geometric features.

4. The parameter determination method according to claim 2, wherein, The feature information corresponding to the image includes pixel information of all or part of the pixels of the image.

5. The parameter determination method according to any one of claims 2-4, wherein, The second sub-device includes a plurality of image acquisition devices. Among them, inputting the feature information corresponding to the image into a pose recognition model to obtain the transformation parameters between the first coordinate system and the second coordinate system based on the pose recognition model includes: Individually input the feature information corresponding to the images acquired by at least some of the plurality of image acquisition devices into the pose recognition model to obtain candidate transformation parameters corresponding to the at least some image acquisition devices respectively; Process the candidate transformation parameters corresponding to the at least some image acquisition devices respectively to obtain The transformation parameters.

6. The parameter determination method according to claim 5, wherein, The first sub-device includes a plurality of marker points, and the image includes at least some of the plurality of marker points. Among them, processing the candidate transformation parameters corresponding to the at least some image acquisition devices respectively to obtain the transformation parameters includes: For each image acquisition device among the at least some image acquisition devices, determine the measured coordinates of the marker points in the captured image, determine the reprojection coordinates of the marker points based on the corresponding candidate transformation parameters, and use the deviation between the measured coordinates and the reprojection coordinates as the error of the candidate transformation parameters; Select a candidate transformation parameter with the smallest error among the candidate transformation parameters corresponding to the at least some image acquisition devices respectively as the transformation parameters.

7. The parameter determination method according to any one of claims 2-4, wherein, The second sub-device includes a plurality of image acquisition devices. Among them, inputting the feature information corresponding to the image into a pose recognition model to obtain the transformation parameters between the first coordinate system and the second coordinate system based on the pose recognition model includes: Input the features corresponding to the images collected by the multiple image acquisition devices into the pose recognition model to obtain conversion parameters corresponding to the multiple image acquisition devices.

8. The parameter determination method according to any one of claims 2-7, wherein, Based on the relative pose, determine the operating parameters of the audio output device, including: Based on the conversion parameters, determine the coordinate values of the audio output device in the first coordinate system; Based on the coordinate values of the audio output device in the first coordinate system, determine the operating parameters of the audio output device.

9. The parameter determination method according to claim 8, wherein, Based on the conversion parameters, determine the coordinate values of the audio output device in the first coordinate system, including: Based on the conversion parameters and the coordinates of the audio output device in the second coordinate system, determine the coordinate values of the audio output device in the first coordinate system.

10. The parameter determination method according to claim 8, wherein, Based on the conversion parameters, determine the coordinate values of the audio output device in the first coordinate system, including: Based on the conversion parameters determined at the first moment and the coordinates of the audio output device in the second coordinate system Determine the coordinate measurement value of the audio output device in the first coordinate system at the first moment; Based on the historical coordinate measurement values obtained during a period of time before the first moment, determine the coordinate prediction value of the audio output device in the first coordinate system at the first moment, where the historical coordinate measurement values are the coordinate measurement values of the audio output device in the first coordinate system obtained based on the images acquired during a period of time before the first moment; Fuse the coordinate measurement value and the coordinate prediction value to obtain a fused coordinate value, and use the fused coordinate value as the coordinate value of the audio output device in the first coordinate system at the first moment.

11. The parameter determination method according to claim 8, wherein, Further include: Based on the historical coordinate data before the first moment, determine the coordinate prediction value of the audio output device in the first coordinate system at the first moment, and use the coordinate prediction value as the coordinate value of the audio output device in the first coordinate system at the first moment.

12. The parameter determination method according to claim 10 or 11, wherein, Based on the historical coordinate data before the first moment, determine the coordinate prediction value of the audio output device in the first coordinate system at the first moment, including: Based on the historical coordinate data before the first moment, update the parameters of the Kalman filter to obtain a Kalman filter with updated parameters; Use the Kalman filter with updated parameters to determine the coordinate prediction value of the audio output device in the first coordinate system at the first moment.

13. The parameter determination method according to any one of claims 8-12, wherein, Based on the coordinate values of the audio output device in the first coordinate system, determine the operating parameters of the audio output device, including: When the first coordinate system is constructed based on the wearing part of the first sub-device, based on the conversion parameters between the first coordinate system and the third coordinate system constructed based on the physical points of the first sub-device, determine the coordinate values of the virtual sound source point on the first sub-device in the first coordinate system; In the first coordinate system, based on the coordinate values of the virtual sound source point and the coordinate values of the audio output device, determine the operating parameters of the audio output device.

14. The parameter determination method according to any one of claims 8-12, wherein, Determine the operating parameters of the audio output device based on the coordinate values of the audio output device in the first coordinate system, including: When the first coordinate system is constructed based on the first sub-device, determine the coordinate values of the virtual sound source point on the first sub-device and the coordinate values of the audio output device in the fourth coordinate system based on the conversion parameters between the first coordinate system and the fourth coordinate system constructed based on the wearing position of the first sub-device; In the fourth coordinate system, determine the operating parameters of the audio output device based on the coordinate values of the virtual sound source point and the coordinate values of the audio output device.

15. The parameter determination method according to any one of claims 1-14, wherein The wearing position of the first sub-device is the head, the wearing position of the second sub-device is the neck, and the audio output device is a speaker.

16. A split device, comprising: A first sub-device; A second sub-device, comprising an audio output device and an image acquisition device; Wherein, the first sub-device or the second sub-device comprises a processor, the image acquisition device is configured to acquire an image in the direction of the first sub-device and send the acquired image to the processor, and the processor is configured to determine the relative pose between the first sub-device and the second sub-device based on the image, determine the operating parameters of the audio output device based on the relative pose, and send the operating parameters to the audio output device.

17. An electronic device, comprising: A processor; A memory storing one or more computer program modules; Wherein, the one or more computer program modules are configured to be executed by the processor to implement the parameter determination method according to any one of claims 1-15.

18. A computer-readable storage medium storing non-transitory computer-readable instructions, which can implement the parameter determination method according to any one of claims 1-15 when the non-transitory computer-readable instructions are executed by a computer.

Images