Space display method and device and electronic equipment

Abstract

The embodiment of the invention provides a space display method and device and electronic equipment. The method comprises the following steps: acquiring a space depth image of a space where a user is located, wherein the space depth image comprises depth-of-field information corresponding to each pixel point; space prompt information is generated based on the space depth image, the space prompt information corresponds to at least one pixel point in the space depth image, and the sensing intensity of the space prompt information is associated with depth-of-field information of the corresponding pixel point in the space depth image; and displaying the space prompt information to the user so as to express the environment condition in the space where the user is located through the space prompt information. According to the method, the relative position relation between the user and other objects in the space where the user is located can be concrete displayed to the user from the non-visual angle, so that concrete of the space structure is achieved from the non-visual angle, the user is assisted in establishing overall cognition on the space where the user is located, and potential safety risks brought by other objects in the space where the user is located are reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically to a spatial display method, device, and electronic device. Background Art

[0002] In daily life, the cognition of space mainly relies on vision. After arriving at a place, such as entering a building, or walking to a new square or street, we usually observe the distribution of objects and the flow of people in the surrounding environment through vision. However, visually impaired people (hereinafter referred to as visually impaired people) cannot observe the surrounding environment through vision, so it is difficult for them to understand the environment they are in, and it is even more difficult for them to carry out daily activities in the environment they are in.

[0003] In order to help the visually impaired understand their environment, the relevant technology uses voice feedback function to give voice reminders when the user approaches an object, so that the user can know the nearby obstacles in advance. However, the visually impaired cannot observe the overall surrounding environment through vision, such as the spatial shape of the surrounding environment, the distribution of objects, the flow of people, etc. Therefore, the visually impaired lacks the ability to predict the potential risks in the surrounding environment.

[0004] Therefore, it is necessary to design a completely new solution to solve the above technical problems. Summary of the invention

[0005] The embodiments of the present application provide an improved method, device and electronic device for displaying space, which are used to provide users with feedback on spatial information at various locations in the space from a non-visual perspective, visualize the spatial structure from a non-visual perspective, help users establish an overall understanding of the space they are in, and reduce safety risks.

[0006] The embodiments of the present application are intended to provide a spatial display method, device and electronic device.

[0007] In a first aspect of the present application, a space display method is provided, comprising:

[0008] Obtaining a spatial depth image of the space where the user is located; the spatial depth image includes depth information corresponding to each pixel point;

[0009] Generate spatial hint information based on the spatial depth image; the spatial hint information corresponds to at least one pixel in the spatial depth image; the perceived intensity of the spatial hint information is associated with the depth of field information of the corresponding pixel in the spatial depth image;

[0010] The spatial prompt information is displayed to the user to indicate the environment conditions in the space where the user is located.

[0011] In some implementations, after obtaining the spatial depth image of the space where the user is located, the method further includes:

[0012] The spatial depth image is mapped to a display interface on the user side to display the spatial depth image to the user.

[0013] In some implementations, generating spatial prompt information based on the spatial depth image includes:

[0014] In response to a touch command on the display interface, obtaining a touch position of the user on the display interface;

[0015] According to the touch position of the user in the display interface, the depth of field information of the pixel point corresponding to the touch position in the spatial depth image is obtained;

[0016] Based on the depth of field information corresponding to the touch position, vibration prompt information corresponding to the touch position is generated; wherein the vibration prompt information is used to indicate the distribution of objects in the space where the user is located; the perceived intensity of the vibration prompt information is determined by at least one of the vibration intensity, the vibration frequency, and the vibration waveform; the perceived intensity of the vibration prompt information changes with the change of the depth of field information corresponding to the touch position;

[0017] Display space prompt information to users, including:

[0018] Based on the perceived intensity of the vibration prompt information, a vibration effect of the vibration prompt information is triggered through a tactile component on the user side; and the vibration effect of the vibration prompt information changes with changes in the perceived intensity.

[0019] In some implementations, generating vibration prompt information corresponding to the touch position based on the depth of field information corresponding to the touch position includes:

[0020] According to the depth of field value corresponding to the touch position, the vibration intensity of the vibration prompt information corresponding to the touch position is set;

[0021] The vibration intensity of the vibration prompt information is inversely proportional to the depth of field value corresponding to the touch position in the spatial depth image.

[0022] In some implementations, after obtaining the spatial depth image of the space where the user is located, the method further includes:

[0023] Perform image semantic segmentation on the spatial depth image to obtain the object corresponding to each pixel in the spatial depth image;

[0024] Generate spatial hint information based on the spatial depth image, including:

[0025] According to the objects corresponding to each pixel point in the spatial depth image and the corresponding relationship between the objects and the obstacle hazard level, vibration warning information matching the obstacle hazard level is set for each pixel point;

[0026] The perceived intensity of the vibration prompt information is determined by at least one of the vibration intensity, the vibration frequency, and the vibration waveform.

[0027] In some embodiments, the vibration frequency of the vibration prompt information is proportional to the obstacle danger level of the corresponding pixel point in the spatial depth image; and / or

[0028] The vibration intensity of the vibration prompt information is proportional to the obstacle danger level of the corresponding pixel point in the spatial depth image.

[0029] In some implementations, generating spatial prompt information based on the spatial depth image includes:

[0030] In response to a voice prompt trigger command issued by the user, a target pixel point is selected based on the depth information of each pixel point in the spatial depth image; each target pixel point corresponds to each object in the space where the user is located; and the difference between the depth value of each target pixel point and the depth value of an adjacent pixel point meets a preset condition;

[0031] Generate voice prompt information corresponding to each target pixel;

[0032] The perceived intensity of the voice prompt information is determined by at least one of a playback order, a playback volume, and a stereo orientation of a sound source.

[0033] In some implementations, displaying spatial prompt information to a user includes:

[0034] According to the depth of field values ​​of each target pixel point in the spatial depth image, the voice prompt information corresponding to each target pixel point is played in sequence at a preset speed; or

[0035] In response to a scanning instruction issued to the space where the user is located, based on the scanning direction and scanning speed included in the scanning instruction, play voice prompt information corresponding to each target pixel point;

[0036] Among them, the playback order of voice prompt information corresponding to each target pixel point is inversely proportional to the depth of field value corresponding to each target pixel point; and / or, the playback volume of the voice prompt information corresponding to each target pixel point is inversely proportional to the depth of field value of each target pixel point; and / or, the stereo orientation of the sound source of the voice prompt information corresponding to each target pixel point is consistent with the position direction of each target pixel point mapped to the user's space.

[0037] In some implementations, obtaining a spatial depth image of the space where the user is located includes:

[0038] Collecting spatial images of the space where the user is located;

[0039] The spatial image is input into the depth prediction model to obtain the depth information corresponding to each pixel in the spatial image;

[0040] A spatial depth image is constructed based on the depth of field information corresponding to each pixel in the spatial image.

[0041] In some implementations, the spatial image is input into a depth prediction model to obtain depth information corresponding to each pixel in the spatial image, including:

[0042] Inputting a plurality of spatial image groups collected from a binocular camera into a first depth feature extraction model to obtain a disparity map of the plurality of spatial image groups; the disparity map includes a depth value of each pixel point; or

[0043] A plurality of continuous spatial images acquired from a monocular camera are input into a second depth feature extraction model to obtain a depth of field value of each pixel point in the plurality of continuous spatial images.

[0044] In some implementations, multiple spatial image groups collected from a binocular camera are input into a first depth feature extraction model to obtain disparity maps of the multiple spatial image groups, including:

[0045] The spatial image group collected by the binocular camera is predicted through a deep neural network to obtain the confidence volume of the spatial image group;

[0046] Calculating the spatial image group to obtain an initial disparity map corresponding to the spatial image group;

[0047] Perform fitting processing on the initial disparity map to obtain an affine transformation corresponding to the initial disparity map;

[0048] Based on affine transformation, the confidence volumes of the double-pixel points in the spatial image group are resampled so that the confidence volumes of the spatial image group are in the same space;

[0049] The confidence volumes of the resampled spatial image group are fused to obtain a low-resolution disparity map of the spatial image group;

[0050] Based on the pixel features of dual-pixel points in the spatial image group, the low-resolution disparity map is subjected to edge-aware refinement processing to obtain a high-resolution disparity map of the spatial image group.

[0051] In some implementations, a plurality of continuous spatial images acquired from a monocular camera are input into a second depth feature extraction model to obtain a depth value of each pixel point in the plurality of continuous spatial images, including:

[0052] Predicting multiple continuous spatial images captured by a monocular camera through a deep neural network to obtain a first depth of field prediction value of each pixel point in the multiple continuous spatial images;

[0053] The optical flow analysis network is used to predict the change trajectory of each pixel point in each spatial image in the next adjacent spatial image, so as to obtain the second depth prediction value of each pixel point in the next adjacent spatial image;

[0054] If the first depth prediction value is consistent with the second depth prediction value, the first depth prediction value is used as the depth value of each pixel point in the plurality of continuous spatial images.

[0055] In some implementations, obtaining a spatial depth image of the space where the user is located includes:

[0056] The depth camera is used to collect spatial depth images from the user's space.

[0057] In a second aspect of the present application, a space display device is provided, the device comprising:

[0058] An acquisition module is used to acquire a spatial depth image of the space where the user is located; the spatial depth image includes depth information corresponding to each pixel point;

[0059] A generating module, configured to generate spatial prompt information based on the spatial depth image; the spatial prompt information corresponds to at least one pixel point in the spatial depth image; and the perceived intensity of the spatial prompt information is associated with the depth of field information of the corresponding pixel point in the spatial depth image;

[0060] The display module is used to display space prompt information to the user, so as to indicate the environment situation in the space where the user is located through the space prompt information.

[0061] In a third aspect of the present application, a computer-readable storage medium is provided, which includes instructions, and when the instructions are executed on a computer, the computer executes the spatial display method described in the first aspect.

[0062] In a fourth aspect of the present application, a computing device is provided, which is configured as: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the spatial display method described in the first aspect when executing the computer program.

[0063] In the technical solution provided by the embodiment of the present application, first, it is necessary to obtain a spatial depth image of the space where the user is located. The spatial depth image includes the depth information corresponding to each pixel point, which provides a data basis for the subsequent generation process of spatial prompt information. Then, spatial prompt information is generated based on the spatial depth image, and the spatial prompt information is displayed to the user, so as to represent the environmental conditions in the space where the user is located through the spatial prompt information. Since the spatial prompt information corresponds to at least one pixel point in the spatial depth image, and the perceptual intensity of the spatial prompt information is associated with the depth information of the corresponding pixel point in the spatial depth image, the user can recognize the relative position relationship between himself and other objects in the space from a non-visual perspective through the depth information reflected in the spatial prompt information display process, thereby realizing the concretization of the spatial structure from a non-visual perspective, assisting the user to establish an overall cognition of the space where he is located, and reducing the potential safety risks brought by other objects in the space. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0065] Figure 1 A schematic diagram of a process flow of the spatial display method in the present application is shown schematically;

[0066] Figure 2 A schematic diagram of a principle of the image construction method in the present application is schematically shown;

[0067] Figure 3 A schematic diagram of a principle of the spatial display method in the present application is schematically shown;

[0068] Figure 4 A schematic diagram of a principle of a prompt generation method according to the present application is schematically shown;

[0069] Figure 5 A schematic diagram of a structure of a space display device according to the present application is shown schematically;

[0070] Figure 6 A schematic diagram of a structure of a computing device according to the present application is shown schematically;

[0071] Figure 7 A structural diagram of a server according to the present application is schematically shown.

[0072] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0073] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0074] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, equipment, system or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0075] In daily life, the cognition of space mainly relies on vision. Generally speaking, after arriving at a place, such as entering a building, or walking to a new square or street, we will observe the distribution of objects and the flow of people in the surrounding environment through vision. However, the visually impaired cannot observe the surrounding environment through vision, so it is difficult for them to understand the environment they are in, and it is even more difficult for them to carry out daily activities in the environment they are in.

[0076] In related technologies, in order to help the visually impaired understand their surroundings, voice feedback functions are used to provide voice reminders when the user approaches an object. Although this prompting method can enable the visually impaired to know obstacles that are close to them in the surrounding environment, the visually impaired cannot observe the overall surrounding environment through vision, such as the spatial shape of the surrounding environment, the distribution of objects, the flow of people, etc. Therefore, the visually impaired lacks the ability to predict potential risks in the surrounding environment.

[0077] In addition, in the related art, the user is also fed back the appearance of nearby objects in the surrounding environment by means of vibration of the handle. However, this method not only fails to overcome the technical problems consistent with the above, but also requires the addition of auxiliary equipment, which increases the cost of hardware modification.

[0078] In summary, a new solution needs to be designed to solve the above technical problems.

[0079] In order to overcome the above technical problems, according to the embodiments of the present application, a space display method, device and electronic device are proposed.

[0080] The applicant has found that compared with the technical solution in the related art that only prompts close objects, the technical solution in the embodiment of the present application, through the correlation between the spatial prompt information and the depth of field information, can also feedback the spatial information of various places in the space to the user from a non-visual perspective, overcoming the spatial cognition difficulties and safety risks caused by the lack of information in the related art, and helping users to establish an overall cognition of the space they are in. Specifically, first, a spatial depth image of the space where the user is located is obtained. The spatial depth image includes the depth of field information corresponding to each pixel point, which provides a data basis for the subsequent generation process of spatial prompt information. Then, based on the spatial depth image, spatial prompt information is generated, and the spatial prompt information is displayed to the user, so as to indicate the environmental conditions in the space where the user is located through the spatial prompt information. Since the spatial prompt information corresponds to at least one pixel point in the spatial depth image, and the perceived intensity of the spatial prompt information is associated with the depth information of the corresponding pixel point in the spatial depth image, therefore, through the depth information reflected in the spatial prompt information display process, the user can recognize the relative position relationship between himself and other objects in the space from a non-visual perspective, assisting the user to establish an overall cognition of the space in which he is located, and reducing the potential safety risks brought by other objects in the space.

[0081] As an optional implementation, the number of spatial display devices may be one or more. The spatial display device may be implemented as a human-computer interaction system or a human-computer interaction program deployed in various devices, and this application does not limit this. For example, the spatial display device may be a hardware circuit disposed in a processing device of various devices (such as a terminal device or a server), or a software module disposed in various applications.

[0082] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.

[0083] The following is a reference to the specific application scenario. Figure 1 To describe the method for spatial display according to the exemplary embodiment of the present application. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0084] The execution process of the space display method is described below in conjunction with the following embodiments. Figure 1 Flow chart of a space display method provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0085] Step 101: Acquire a spatial depth image of the space where the user is located.

[0086] The space mentioned in the embodiments of the present application can be any building space, such as a factory, a room in a commercial housing, an office, etc. The building space can be regarded as consisting of a ceiling, a floor and a wall. Alternatively, the space can also be an open space, such as a square, a street, or nature.

[0087] In the embodiment of the present application, the spatial depth image carries the depth information of each object in the image. It is understandable that the depth information of each object is mainly used to indicate the relative position relationship between the image acquisition device and each object. For example, the relative distance between the image acquisition device and each object is used as the depth value of each object.

[0088] In practical applications, various objects include but are not limited to: objects, vehicles, people, animals, plants, and building structures in the space. For a space, the building structure is relatively fixed, and the depth of field information between the building structure helps to quickly understand the basic structure of the space. However, the building structure varies with the type of space. For example, if the space where the current user is located is a factory building, then the building structure includes walls, floors, ceilings, columns, windows, stairs, doors, etc. For example, if the space where the current user is located is a square, then the building structure includes the ground, steps on the square, fences, railings, shops, etc. When collecting spatial depth images, the main focus is on spatial shapes (including fixed building structures in the space) to assist users in quickly understanding the basic structure of the space they are in. In some optional embodiments, when collecting spatial depth images, objects with mobility and temporary nature, such as people, animals, vehicles, plants, and other movable items in the space, are eliminated to avoid confusion when users (especially visually impaired people) perceive the current space.

[0089] Of course, in some optional embodiments, in addition to the shape of the space, people, animals, vehicles, plants and other movable items in the space can also be used as part of the space to provide a data basis for the subsequent acquisition of spatial information, enrich the details of the spatial information, and ensure the integrity and real-time nature of the spatial information. Furthermore, such objects are marked and displayed using different types of prompt information to assist users in accurately distinguishing mobile and temporary objects in the space and reduce safety risks.

[0090] As an optional embodiment, in step 101, obtaining a spatial depth image of the space where the user is located may be implemented by: collecting the spatial depth image from the space where the user is located by using a depth-of-field camera.

[0091] For example, the depth of field camera can be a depth camera component installed in a terminal device, or a camera for taking spatial depth images, or other devices or modules with spatial depth image acquisition functions.

[0092] For example, a depth image of a certain space is captured by a depth camera component in a mobile phone that can capture a spatial depth image, thereby obtaining the above-mentioned spatial depth image. In practical applications, one spatial depth image can be captured for one space. Of course, if the area of ​​the space is large, multiple spatial depth images can also be captured. The subsequent processing of each spatial depth image is the same.

[0093] For example, a laser radar (Light Detection and Ranging, LiDAR) is used to directly obtain a more accurate depth of field image. Among them, LiDAR is used to obtain point cloud data and generate an accurate digital three-dimensional model, thereby obtaining a three-dimensional real scene of the surrounding environment.

[0094] As an alternative embodiment, see Figure 2 In step 101, obtaining a spatial depth image of the space where the user is located can be implemented as follows:

[0095] Step 201, collecting a spatial image of the space where the user is located;

[0096] Step 202, inputting the spatial image into a depth of field prediction model to obtain depth of field information corresponding to each pixel point in the spatial image;

[0097] Step 203: construct a spatial depth image based on the depth information corresponding to each pixel in the spatial image.

[0098] Specifically, the spatial image can be understood as an image captured for part or all of the area in the space where the user is located. The difference from the spatial depth image is that the spatial image does not have the depth information of each object in the space.

[0099] In step 201, taking the use of the camera in a mobile phone as a shooting device as an example, an image of a certain space is shot by a camera component in the mobile phone that can shoot spatial images, thereby obtaining the above-mentioned spatial image. In practical applications, one or more images can be shot in a space. The preprocessing of each image is the same. In order to facilitate three-dimensional reconstruction in subsequent processing, overlapping areas are required in multiple images. In addition, in fact, a video picture of the space can also be shot using a mobile phone, the video picture is segmented into image frames, and the obtained image is subsequently processed.

[0100] After taking a spatial image of a space, in order to create a spatial depth image based on the spatial image, the shape of the space needs to be known based on the spatial image. The depth information of each object in the space can just reflect the shape of the space. Therefore, the panorama needs to be analyzed first to determine the depth information of each object in the space. For example, the distance from the user to objects such as tables, chairs, cabinets, and walls is the depth value of these objects.

[0101] Assume that the shooting device of the spatial image is a binocular camera. Based on this, in an optional implementation of step 202, multiple spatial image groups collected from the binocular camera are input into the first depth feature extraction model to obtain disparity maps of multiple spatial image groups. Among them, the disparity map includes the depth of field value of each pixel point. Further optionally, the first depth feature extraction model can be implemented as a Du2Net model. The Du2Net model is a depth estimation network model based on deep learning, which combines the stereo information of dual cameras and dual pixels to predict and process the spatial image group.

[0102] Specifically, in one example, in step 202, the spatial image group collected by the binocular camera is predicted by a deep neural network to obtain the confidence volume of the spatial image group. Among them, the spatial image group is composed of images collected independently by two cameras. Each spatial image group may include a pair of spatial images captured at the same time, and the pair of spatial images is two spatial images captured by two cameras at the same time of the current space. It can also include spatial images captured by two cameras at different times. Then, the spatial image group is calculated to obtain the initial disparity map corresponding to the spatial image group. Specifically, multiple photos collected by the dual cameras in the mobile phone and the dual pixel data in the photos corresponding to each camera can be used to calculate the initial disparity map corresponding to the dual cameras and dual pixels in the spatial image group. Then the initial disparity map is fitted to obtain the affine transformation corresponding to the initial disparity map. Then, based on the affine transformation corresponding to the initial disparity map, the confidence volume of the dual pixel points in the spatial image group is resampled so that the confidence volume of the spatial image group is in the same space. It should be noted that the dual pixel points in the spatial image group are collected by the same camera from the position points in the space where the user is located and have a mapping relationship. Then, the confidence volumes of the resampled spatial image group are fused to obtain a low-resolution disparity map of the spatial image group. Finally, based on the pixel features of the dual pixel points in the spatial image group, the low-resolution disparity map is edge-aware refined to obtain a high-resolution disparity map of the spatial image group. In this way, the two-dimensional spatial image collected by the binocular camera can be converted into a three-dimensional high-resolution disparity map, thereby providing depth of field information in the space for subsequent processing, simplifying the acquisition process of the spatial depth image, reducing the difficulty of acquiring the spatial depth image, and reducing the cost of equipment modification.

[0103] In another optional implementation of step 202, a plurality of continuous spatial images acquired from a monocular camera are input into a second depth feature extraction model to obtain a depth of field value of each pixel point in the plurality of continuous spatial images.

[0104] Further optionally, the second depth feature extraction model can be implemented as a Monodepth2 model. The Monodepth2 model is an unsupervised learning network model that estimates depth through the consistency of optical flow and depth of field continuity between images. This model does not require depth labels for learning, but only requires multiple images captured by the camera in a continuous time for learning. In simple terms, it uses a deep neural network for prediction, then uses an optical flow network to predict the movement of objects, and finally combines the prediction results of the two networks and optimizes them using the original image reconstruction loss.

[0105] Specifically, in one example, in step 202, a plurality of continuous spatial images captured by a monocular camera are predicted by a deep neural network to obtain a first depth prediction value of each pixel in the plurality of continuous spatial images. Further, the change trajectory of each pixel in each spatial image in the next adjacent spatial image is predicted by an optical flow analysis network to obtain a second depth prediction value of each pixel in the next adjacent spatial image. Since the only solution that can keep the image optical flow prediction value consistent is the actual depth of field of the image, if the first depth prediction value is consistent with the second depth prediction value, the first depth prediction value of the deep neural network is the same as the second depth prediction value obtained by the optical flow analysis network based on the continuous change trajectory, that is, the second depth prediction value is the only solution consistent with the first depth prediction value. In this case, the first depth prediction value is used as the depth value of each pixel in the plurality of continuous spatial images.

[0106] Finally, in step 203, the depth information is added to each pixel in the spatial image to obtain a spatial depth image. For example, the converted depth value (or the depth information in the high-resolution disparity map) is added to the z-axis corresponding to each pixel as the z-axis coordinate, and the original two-dimensional image coordinates of each pixel are used as the x-axis and y-axis coordinates, thereby constructing a spatial depth image containing the depth value.

[0107] Step 102: Generate spatial prompt information based on the spatial depth image.

[0108] Step 103: displaying space prompt information to the user, so as to indicate the environment of the space where the user is located through the space prompt information.

[0109] In an embodiment of the present application, the spatial prompt information corresponds to at least one pixel in the spatial depth image. In fact, in order to simplify the content of the feedback information, in an optional embodiment, the target object corresponding to each pixel in the spatial depth image is further identified, and the spatial prompt information is generated based on the target object. For example, for the same object, a unified spatial prompt information is generated. For example, the spatial prompt information stored in all the pixels corresponding to a table is consistent.

[0110] Of course, in actual applications, if multiple pixels correspond to a target object, but the target object is large in size, and there are certain differences in the relative positions of the various parts of the target object and the user, in this case, in another optional embodiment, different spatial prompt information can also be generated for the same object. For example, for multiple pixels corresponding to a moving vehicle, different spatial prompt information corresponding to each pixel is generated based on the depth of field information of each pixel. Taking the vibration prompt information as an example, the vibration intensity corresponding to the part closer to the user is greater, and the vibration intensity corresponding to the part farther from the user is smaller.

[0111] In addition to the above two optional embodiments, the correspondence between the spatial prompt information and each pixel in the spatial depth image may also be set to other forms, which are not limited in this embodiment.

[0112] As an optional implementation, after step 101, the spatial depth image can also be mapped to the display interface of the user side to display the spatial depth image to the user. For example, the user side can be set in a dialogue system, and a chat robot can communicate with the user by voice to trigger an automated spatial information feedback process for the current space for the user, thereby improving the convenience of user operation and the efficiency of spatial information feedback. Further optionally, the color of the pixel in the spatial depth image is related to the corresponding depth of field value in the spatial depth image. For example, the depth of field value of the pixel in the spatial image is represented by a grayscale value, that is, different grayscale values ​​are used to represent pixels belonging to different depth of field value ranges.

[0113] Further optionally, different modules are used to replace each object in the spatial depth image, so that the user can select and obtain the spatial prompt information of the selected object. Figure 3 Taking the interface shown in the figure as an example, the left part shows the original interface containing the spatial image, and the right part shows the spatial prompt interface, which contains the spatial depth image corresponding to the spatial image. Figure 3 In the space prompt interface, the furniture in the user's space is represented by model icons with different grayscale values. After touching different model icons in the space prompt interface, the vibration prompt information corresponding to the model icon will be synchronously fed back. The perceived intensity of the vibration prompt information is related to the depth of field value of the object corresponding to the model icon. For example, touching the mirror icon in the space prompt interface will feedback vibration prompt information that matches the depth of field of the mirror. Touching the chair icon in the space prompt interface will feedback vibration prompt information that matches the depth of field of the chair. Compared with the mirror, the chair is closer to the user, and the corresponding vibration prompt information has a stronger touch.

[0114] In an embodiment of the present application, the perceived intensity of the spatial prompt information is associated with the depth of field information of the corresponding pixel point in the spatial depth image. It can be understood that the perceived intensity of the spatial prompt information refers to the intensity of the somatosensory experience that the user can obtain. For different types of spatial prompt information, as the spatial prompt information differs in the display form, the presentation method of the change in perceived intensity also differs. For example, the change in the perceived intensity of the vibration prompt information presented in the form of vibration can be reflected as a change in one or more aspects of the vibration intensity, vibration waveform, vibration frequency, and vibration application position. For example, the change in the perceived intensity of the voice prompt information can be reflected in one or more aspects of the playback order, playback volume, timbre, and stereo orientation of the sound source of the voice prompt information. It can be seen that the different perceived intensities of the spatial prompt information represent the differences in the spatial distribution of various objects in the space where the user is located, so that the user can intuitively perceive the spatial position relationship between the objects corresponding to the pixel points in the spatial depth image and the overall spatial structure of the space from a non-visual perspective, thereby realizing the concretization of the spatial structure from a non-visual perspective.

[0115] The following describes the generation process and display form of spatial prompt information in combination with different types of spatial prompt information.

[0116] Assume that the spatial prompt information is a vibration prompt information. Based on the display interface provided in the above implementation, see Figure 4 In step 102, the spatial prompt information is generated based on the spatial depth image, which can be implemented as follows:

[0117] Step 401, in response to a touch command on a display interface, obtaining a touch position of a user on the display interface;

[0118] Step 402, according to the touch position of the user in the display interface, obtaining the depth information of the pixel point corresponding to the touch position in the spatial depth image;

[0119] Step 403: Generate vibration prompt information corresponding to the touch position based on the depth of field information corresponding to the touch position.

[0120] Among them, the vibration prompt information is used to indicate the distribution of objects in the space where the user is located. The perceived intensity of the vibration prompt information is determined by at least one of the vibration intensity, the vibration frequency, and the vibration waveform. Compared with a single vibration prompt method, the vibration prompt information acquisition method in the above steps 401 to 403 can convert the depth of field of the object corresponding to the touch position into the vibration prompt information of the corresponding perception intensity, so that more spatial position relationships are reflected in the vibration prompt information fed back to the user in the form of vibration touch. In the embodiment of the present application, the perceived intensity of the vibration prompt information changes with the change of the depth of field information of the corresponding touch position. For example, different perception intensities corresponding to different depth of field values ​​are configured, so that the user can intuitively perceive the changes in the spatial structure constructed by different depth of field values ​​through the vibration touch brought by the vibration prompt information. For example, different forms of vibration waveforms can also be configured for different types of objects. For example, if the threat of a moving object is relatively large, a vibration waveform with a sharper waveform change trend can be configured for such an object; if the threat of a fixed object is relatively small, a vibration waveform with a relatively gentle waveform change trend can be configured for such an object. Therefore, the change in the perceived intensity of the vibration prompt information can show the user the change in the positional relationship between the user and various objects in the space from a tactile perspective, making the spatial distribution three-dimensionally perceptible to the user.

[0121] After the vibration prompt information is generated, the spatial prompt information is displayed to the user in step 103, which can be implemented as follows: based on the perceived intensity of the vibration prompt information, the vibration effect of the vibration prompt information is triggered by the tactile component on the user side. Among them, the vibration effect of the vibration prompt information changes with the change of the perceived intensity. Optionally, the closer the distance, the stronger the perceived intensity of the vibration prompt information, thereby providing the user with a concrete distribution of spatial objects through a tactile manner. For example, the closer the object is to the user, the stronger the perceived intensity of the vibration prompt information corresponding to the object. Specifically, the closer the object is to the user, when the user touches the object area in the spatial depth image, the stronger the vibration intensity of the vibration prompt information fed back to the user through the tactile component, and the higher the vibration frequency.

[0122] For example, after the spatial depth image is mapped to the device screen (i.e., display interface) on the user side, the user touches the spatial depth image displayed on the device screen. In step 401, in response to the touch operation instruction on the device screen, the touch trajectory of the user on the device screen is identified. In step 402, the depth of field value of the pixel point corresponding to the touch trajectory in the spatial depth image is obtained according to the touch trajectory of the user. Furthermore, in step 403, based on the depth of field value of these pixel points, the vibration intensity of the vibration prompt information corresponding to these pixel points is generated. In an optional embodiment, step 403 can be implemented as follows: according to the depth of field value corresponding to the touch position, the vibration intensity of the vibration prompt information corresponding to the touch position is set. Among them, the vibration intensity of the vibration prompt information is inversely proportional to the depth of field value corresponding to the touch position in the spatial depth image. For example, the larger the depth of field value of an object, the farther the object is from the user. In this case, the vibration intensity and vibration frequency assigned to the object are lower. Conversely, the smaller the depth of field value of an object, the closer the object is to the user. In this case, the vibration intensity and vibration frequency assigned to the object are higher, thereby achieving a tactile sensation in which nearby objects are more intense and the vibration decreases from strong to weak as objects move from near to far.

[0123] Based on the above example, in step 103, the corresponding vibration prompt information will be synchronously fed back along the touch track of the user on the device screen, so that the user can feel the vibration touch consistent with the touch track, and thus know the current spatial structure based on the vibration touch of different vibration intensities. In practical applications, taking the IOS platform as an example, the vibration prompt information can be implemented using CoreHaptics. Specifically, CHHapticPattern can be used to adjust the vibration intensity, or .hapticIntensity can be used as an adjustment tool.

[0124] Of course, in practical applications, different vibration waveforms can be used to indicate the distance between an object and a user. For example, in an indoor scene, the vibration waveform corresponding to an object within 3 meters from the user is waveform a, the vibration waveform corresponding to an object more than 3 meters and less than 10 meters from the user is waveform b, and the vibration waveform corresponding to an object more than 10 meters from the user is waveform c. Furthermore, when a user touches an object within different distance intervals, he or she can obtain vibration prompt information of different vibration waveforms, thereby more concretely perceiving the distribution of objects in the space in which he or she is located through waveforms of different tactile sensations.

[0125] In order to further enhance the spatial sense of the vibration prompt information, it is further optional to shield objects within a preset distance interval, and adjust the objects to be displayed and the corresponding vibration prompt information according to the user's moving trajectory, thereby strengthening the user's vertical spatial cognition from a tactile perspective. That is, from a tactile perspective, a spatial display effect of changing scenery is provided to the user.

[0126] In another optional implementation, assuming that the spatial prompt information is voice prompt information, generating the spatial prompt information based on the spatial depth image in step 102 can be implemented as follows: in response to a voice prompt trigger instruction issued by the user, selecting a target pixel point based on the depth information of each pixel point in the spatial depth image; and generating voice prompt information corresponding to each target pixel point.

[0127] Among them, each target pixel corresponds to each object in the space where the user is located. The difference between the depth of field value of each target pixel and the depth of field value of the adjacent pixel meets the preset conditions. It can be understood that in some optional implementations, the difference between the depth of field value of the target pixel and the depth of field value of the adjacent pixel is less than the set threshold. In this case, it means that the target pixel and the adjacent pixel belong to the same object surface. Therefore, in order to avoid repeated broadcasting of voice prompt information and reduce the amount of voice information broadcast, in an optional implementation, each pixel in the spatial depth image can be traversed, and the target pixel corresponding to each object can be screened out from the spatial depth image based on the depth of field information, and the other pixel adjacent to each target pixel can be deduplicated. In practical applications, in addition to the above difference judgment based on the depth of field value, other technical means such as image segmentation and semantic recognition can also be used to achieve deduplication, which will not be elaborated here.

[0128] In the implementation manner of the present application, the perceived intensity of the voice prompt information is determined by at least one of the playback order, the playback volume, and the stereo orientation of the sound source. In the embodiment of the present application, the playback order of the voice prompt information corresponding to each target pixel is inversely proportional to the depth of field value of each target pixel in the spatial depth image. For example, the smaller the depth of field value of an object, the closer the object is to the user. In this case, the playback order of the voice prompt information is higher; conversely, the larger the depth of field value of an object, the farther the object is from the user. In this case, the playback order of the voice prompt information is higher.

[0129] The playback volume of the voice prompt information corresponding to each target pixel is inversely proportional to the depth of field value of the corresponding target pixel in the spatial depth image. For example, the smaller the depth of field value of an object, the closer the object is to the user. In this case, the playback volume of the voice prompt information is louder; conversely, the larger the depth of field value of an object, the farther the object is from the user. In this case, the playback volume of the voice prompt information is lower, thereby achieving the auditory experience that the closer the object is, the louder the sound.

[0130] The stereo orientation of the sound source of the voice prompt information corresponding to each target pixel is consistent with the position direction of each target pixel mapped to the user's space. For example, an object is in a certain position in the space. In this case, the stereo orientation of the sound source of the voice prompt information corresponding to the object is consistent with the certain position. If the user moves in the space, or changes the body posture (such as turning the head, turning around, etc.), the stereo orientation of the sound source of the voice prompt information corresponding to the object will change with the user's movement, so that the stereo orientation of the sound source of the voice prompt information corresponding to the object corresponds to the relative position between the user and the object. Thereby, it helps users better judge obstacles in the space and solves the error problem between the hearing and the camera after the user turns his head.

[0131] Compared with the traditional voice broadcasting method, this method can not only reflect the spatial position relationship between the object and the user through the playback order and the stereo orientation of the sound source, but also reflect the change in the distance between the object and the user through changes in the playback volume and timbre, and even reflect other information such as the type of object and the degree of danger of the object. In this way, through the changes in the perceived intensity of voice prompt information, the distribution of objects in the space where the user is located can be shown to the user in a more three-dimensional and intuitive way from an auditory perspective.

[0132] Based on the above implementation, before step 103, the currently enabled voice prompt mode is determined. In practical applications, the voice prompt mode includes but is not limited to: voice broadcast mode and scanning mode. The above two modes can be implemented independently or in collaboration.

[0133] If the current mode is voice broadcast mode, in step 103, voice prompt information corresponding to each target pixel point is played in sequence at a preset speed according to the depth of field value of each target pixel point in the spatial depth image from small to large.

[0134] As an optional embodiment, all pixels are traversed from the spatial depth image to find the target pixel whose depth value ranks in the top preset position. Optionally, it is detected whether the current pixel is in a preset range of pixels where there are adjacent pixels whose depth value ranks in the top preset position. If there are adjacent pixels whose depth value ranks in the top preset position, it is determined not to use the current pixel as the target pixel. For example, assuming that the depth value is stored in the z-axis coordinate of the pixel, then all pixels are traversed from the spatial depth image to obtain the target pixel whose z value ranks in the top 5 and is in the non-adjacent area.

[0135] Based on the above example, in step 103, in the voice broadcast mode, if the target pixel points with the top z values ​​and non-adjacent are found, the coordinates of the corresponding pixel points are converted into the stereoscopic coordinates of the sound source according to the order of z values ​​from small to large.

[0136] In actual applications, taking the IOS platform as an example, create an AVAudioEngine object as the output device for voice prompt information. Create an AVAudioPlayerNode object and use the new rendering mode in AVAudioEnvironmentNode to automatically select the best spatial audio rendering algorithm according to the output device. Then, attach the node to the engine and use EnvironmentNode to specify the listener position. After initializing the AVAudioPlayerNode object to AVAudio3DMixing, you can adjust the position and set the rendering algorithm to the Head Related Transfer Function (HRTF), and finally play the voice prompt audio of the spatial information. Among them, AVAudio3DPoint contains three parameters xyz. The (x, y) value of the plane is divided by the preset value, and then the depth of field value is directly added to the z value to realize the conversion from two-dimensional coordinates to the stereoscopic orientation of the sound source.

[0137] If the current mode is scanning, in step 103, in response to a scanning instruction issued to the space where the user is located, voice prompt information corresponding to each pixel point is played based on the scanning direction and scanning speed included in the scanning instruction.

[0138] In another optional implementation, after obtaining the spatial depth image of the space where the user is located in step 101, the spatial depth image can also be subjected to image semantic segmentation to obtain the objects corresponding to each pixel in the spatial depth image. Based on this, in step 102, generating spatial prompt information based on the spatial depth image can be implemented as follows: according to the objects corresponding to each target pixel in the spatial depth image, and the correspondence between the objects and the obstacle hazard level, vibration warning information matching the obstacle hazard level is set for each target pixel. Among them, the perceived intensity of the vibration warning information is determined by at least one of the vibration intensity, vibration frequency, and vibration waveform. Here, the method of obtaining the target pixel is similar to the method of obtaining the target pixel in the previous implementation, and will not be elaborated here.

[0139] In the embodiment of the present application, the vibration frequency of the vibration warning information is proportional to the obstacle danger level of the corresponding pixel point in the spatial depth image. And / or, the vibration intensity of the vibration warning information is proportional to the obstacle danger level of the corresponding pixel point in the spatial depth image.

[0140] Through the above-mentioned optional implementation manner, the vibration touch can be provided to the user through the vibration warning information to feedback the danger level of the object. For example, collisions may occur with moving vehicles, people, animals, etc., and the corresponding danger level is relatively high. Therefore, a stronger vibration warning information can be used to indicate the presence of these dangerous objects in the user's current space.

[0141] Further optionally, in addition to depth of field information, risk factors such as temperature and item type (for example, whether it carries a hazardous goods mark or whether it is fragile) can be used to set vibration warning information or voice warning information that matches the above risk factors for the target pixel points, so as to prompt the user that there are dangerous or potentially dangerous objects in the space where he is located, thereby further assisting the user in establishing an overall understanding of the space where he is located and reducing safety risks.

[0142] In an embodiment of the present application, first, it is necessary to obtain a spatial depth image of the space where the user is located. The spatial depth image includes depth information corresponding to each pixel point, providing a data basis for the subsequent generation process of spatial prompt information. Then, spatial prompt information is generated based on the spatial depth image, and the spatial prompt information is displayed to the user, so as to represent the environmental conditions in the space where the user is located through the spatial prompt information. Since the spatial prompt information corresponds to at least one pixel point in the spatial depth image, and the perceived intensity of the spatial prompt information is associated with the depth information of the corresponding pixel point in the spatial depth image, the user can recognize the relative position relationship between himself and other objects in the space from a non-visual perspective through the depth information reflected in the spatial prompt information display process, thereby realizing the concretization of the spatial structure from a non-visual perspective, assisting the user to establish an overall cognition of the space where he is located, and reducing the potential safety risks brought by other objects in the space.

[0143] After introducing the method of the embodiment of the present application, next, refer to Figure 5 The space display device of the embodiment of the present application is introduced.

[0144] The space display device 50 in the embodiment of the present application can realize the above Figure 1 The steps of the spatial display method in the corresponding embodiment. The functions implemented by the spatial display device 50 can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. The spatial display device 50 is applied to a server device or a terminal device. The spatial display device 50 may include an acquisition module 501, a generation module 502, and a display module 503. The functional implementation of each module can refer to Figure 1 The operations performed in the corresponding embodiments are not described in detail here.

[0145] In some implementations, the acquisition module 501 is used to acquire a spatial depth image of the space where the user is located; the spatial depth image includes depth information corresponding to each pixel point;

[0146] A generating module 502 is used to generate spatial hint information based on the spatial depth image; the spatial hint information corresponds to at least one pixel in the spatial depth image; and the perceived intensity of the spatial hint information is associated with the depth of field information of the corresponding pixel in the spatial depth image;

[0147] The display module 503 is used to display space prompt information to the user, so as to indicate the environment of the space where the user is located through the space prompt information.

[0148] In some implementations, the display module 503 is further configured to:

[0149] After the acquisition module 501 acquires the spatial depth image of the space where the user is located, the spatial depth image is mapped to the display interface on the user side to display the spatial depth image to the user.

[0150] In some implementations, when the generating module 502 generates the spatial prompt information based on the spatial depth image, it is configured to:

[0151] In response to a touch command on the display interface, obtaining a touch position of the user on the display interface;

[0152] According to the touch position of the user in the display interface, the depth of field information of the pixel point corresponding to the touch position in the spatial depth image is obtained;

[0153] Based on the depth of field information corresponding to the touch position, vibration prompt information corresponding to the touch position is generated; wherein the vibration prompt information is used to indicate the distribution of objects in the space where the user is located; the perceived intensity of the vibration prompt information is determined by at least one of the vibration intensity, the vibration frequency, and the vibration waveform; the perceived intensity of the vibration prompt information changes with the change of the depth of field information corresponding to the touch position;

[0154] When displaying space prompt information to the user, the display module 503 is configured to:

[0155] Based on the perceived intensity of the vibration prompt information, a vibration effect of the vibration prompt information is triggered through a tactile component on the user side; and the vibration effect of the vibration prompt information changes with changes in the perceived intensity.

[0156] In some implementations, when the generating module 502 generates the vibration prompt information corresponding to the touch position based on the depth information corresponding to the touch position, it is configured as follows:

[0157] According to the depth of field value corresponding to the touch position, the vibration intensity of the vibration prompt information corresponding to the touch position is set;

[0158] The vibration intensity of the vibration prompt information is inversely proportional to the depth of field value corresponding to the touch position in the spatial depth image.

[0159] In some implementations, the apparatus further comprises a semantic segmentation module configured to:

[0160] After the acquisition module 501 acquires the spatial depth image of the space where the user is located, image semantic segmentation is performed on the spatial depth image to obtain objects corresponding to each pixel in the spatial depth image.

[0161] When the generating module 502 generates the spatial prompt information based on the spatial depth image, it is configured to:

[0162] According to the objects corresponding to each pixel point in the spatial depth image and the corresponding relationship between the objects and the obstacle hazard level, vibration warning information matching the obstacle hazard level is set for each pixel point;

[0163] The perceived intensity of the vibration prompt information is determined by at least one of the vibration intensity, the vibration frequency, and the vibration waveform.

[0164] In some embodiments, the vibration frequency of the vibration prompt information is proportional to the obstacle hazard level of the corresponding pixel point in the spatial depth image; and / or, the vibration intensity of the vibration prompt information is proportional to the obstacle hazard level of the corresponding pixel point in the spatial depth image.

[0165] In some implementations, when the generating module 502 generates the spatial prompt information based on the spatial depth image, it is configured to:

[0166] In response to a voice prompt trigger command issued by the user, a target pixel point is selected based on the depth information of each pixel point in the spatial depth image; each target pixel point corresponds to each object in the space where the user is located; and the difference between the depth value of each target pixel point and the depth value of an adjacent pixel point meets a preset condition;

[0167] Generate voice prompt information corresponding to each target pixel;

[0168] The perceived intensity of the voice prompt information includes at least one of a play order, a play volume, and a stereo orientation of a sound source.

[0169] In some implementations, when the display module 503 displays the space prompt information to the user, it is configured to:

[0170] According to the depth of field values ​​of each target pixel point in the spatial depth image, the voice prompt information corresponding to each target pixel point is played in sequence at a preset speed; or

[0171] In response to a scanning instruction issued to the space where the user is located, based on the scanning direction and scanning speed included in the scanning instruction, play voice prompt information corresponding to each target pixel point;

[0172] Among them, the playback order of voice prompt information corresponding to each target pixel point is inversely proportional to the depth of field value corresponding to each target pixel point; and / or, the playback volume of the voice prompt information corresponding to each target pixel point is inversely proportional to the depth of field value of each target pixel point; and / or, the stereo orientation of the sound source of the voice prompt information corresponding to each target pixel point is consistent with the position direction of each target pixel point mapped to the user's space.

[0173] In some implementations, when the acquisition module 501 acquires the spatial depth image of the space where the user is located, it is configured to:

[0174] Collecting spatial images of the space where the user is located;

[0175] The spatial image is input into the depth prediction model to obtain the depth information corresponding to each pixel in the spatial image;

[0176] A spatial depth image is constructed based on the depth of field information corresponding to each pixel in the spatial image.

[0177] In some implementations, when the acquisition module 501 inputs the spatial image into the depth prediction model to obtain the depth information corresponding to each pixel in the spatial image, it is configured as follows:

[0178] Inputting a plurality of spatial image groups collected from a binocular camera into a first depth feature extraction model to obtain a disparity map of the plurality of spatial image groups; the disparity map includes a depth value of each pixel point; or

[0179] A plurality of continuous spatial images acquired from a monocular camera are input into a second depth feature extraction model to obtain a depth of field value of each pixel point in the plurality of continuous spatial images.

[0180] In some implementations, when the acquisition module 501 inputs a plurality of spatial image groups collected from a binocular camera into a first depth feature extraction model and obtains a disparity map of the plurality of spatial image groups, the acquisition module 501 is configured as follows:

[0181] The spatial image group collected by the binocular camera is predicted through a deep neural network to obtain the confidence volume of the spatial image group;

[0182] Calculating the spatial image group to obtain an initial disparity map corresponding to the spatial image group;

[0183] Perform fitting processing on the initial disparity map to obtain an affine transformation corresponding to the initial disparity map;

[0184] Based on affine transformation, the confidence volumes of the double-pixel points in the spatial image group are resampled so that the confidence volumes of the spatial image group are in the same space;

[0185] The confidence volumes of the resampled spatial image group are fused to obtain a low-resolution disparity map of the spatial image group;

[0186] Based on the pixel features of dual-pixel points in the spatial image group, the low-resolution disparity map is subjected to edge-aware refinement processing to obtain a high-resolution disparity map of the spatial image group.

[0187] In some implementations, when the acquisition module 501 inputs a plurality of continuous spatial images acquired from a monocular camera into a second depth feature extraction model and obtains the depth value of each pixel point in the plurality of continuous spatial images, the acquisition module 501 is configured as follows:

[0188] Predicting multiple continuous spatial images captured by a monocular camera through a deep neural network to obtain a first depth of field prediction value of each pixel point in the multiple continuous spatial images;

[0189] The optical flow analysis network is used to predict the change trajectory of each pixel point in each spatial image in the next adjacent spatial image, so as to obtain the second depth prediction value of each pixel point in the next adjacent spatial image;

[0190] If the first depth prediction value is consistent with the second depth prediction value, the first depth prediction value is used as the depth value of each pixel point in the plurality of continuous spatial images.

[0191] In some implementations, when the acquisition module 501 acquires the spatial depth image of the space where the user is located, it is configured to:

[0192] The depth camera is used to collect spatial depth images from the user's space.

[0193] In an embodiment of the present application, a spatial depth image of the space where the user is located needs to be obtained through a spatial display device. The spatial depth image includes depth information corresponding to each pixel point, providing a data basis for the subsequent generation process of spatial prompt information. Furthermore, spatial prompt information is generated based on the spatial depth image, and the spatial prompt information is displayed to the user, so as to represent the environmental conditions in the space where the user is located through the spatial prompt information. Since the spatial prompt information corresponds to at least one pixel point in the spatial depth image, and the perceptual intensity of the spatial prompt information is associated with the depth information of the corresponding pixel point in the spatial depth image, the user can recognize the relative position relationship between himself and other objects in the space from a non-visual perspective through the depth information reflected in the spatial prompt information display process, thereby realizing the concretization of the spatial structure from a non-visual perspective, assisting the user to establish an overall cognition of the space where he is located, and reducing the potential safety risks brought by other objects in the space.

[0194] After introducing the method and device of the embodiment of the present application, the computer-readable storage medium of the embodiment of the present application is described next. The computer-readable storage medium may be a CD, on which a computer program (i.e., a program product) is stored. When the computer program is executed by the processor, each step recorded in the above method implementation will be implemented, for example, obtaining a spatial depth image of the space where the user is located; the spatial depth image includes depth information corresponding to each pixel; generating spatial prompt information based on the spatial depth image; the spatial prompt information corresponds to at least one pixel in the spatial depth image; the perceived intensity of the spatial prompt information is associated with the depth information of the corresponding pixel in the spatial depth image; and displaying the spatial prompt information to the user, so as to indicate the environmental conditions in the space where the user is located through the spatial prompt information. The specific implementation method of each step will not be repeated here.

[0195] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0196] The above describes the space display device 50 in the embodiment of the present application from the perspective of modular functional entities. The following describes the server and terminal device that execute the space display method in the embodiment of the present application from the perspective of hardware processing.

[0197] It should be noted that in the embodiment of the space display device of the present application Figure 5 The physical devices corresponding to the acquisition module 501 and the display module 503 shown may be an input / output unit, a transceiver, a radio frequency circuit, a communication module, an input / output (I / O) interface, etc., and the physical device corresponding to the generation module 502 may be a processor. Figure 5 The space display device 50 shown may have Figure 6 The structure shown, when Figure 5 The space display device 50 shown has Figure 6 When the structure shown is Figure 6 The processor and transceiver in the device can realize the same or similar functions as those in the various modules provided in the device embodiment corresponding to the device. Figure 6 The memory in the storage unit stores a computer program that needs to be called when the processor executes the above-mentioned spatial display method.

[0198] Figure 71 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1122 (for example, one or more processors) and memory 1132, and one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1122 can be configured to communicate with the storage medium 1130 and execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0199] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0200] The steps performed by the server in the above embodiment can be based on the Figure 7 The structure of the server 1100 shown in FIG. Figure 7 The steps performed by the spatial display device 80 shown can be based on the Figure 7 For example, the central processor 1122 performs the following operations by calling the instructions in the memory 1132:

[0201] A spatial depth image of the space where the user is located is obtained through the input and output interface 1158; the spatial depth image includes depth information corresponding to each pixel; spatial prompt information is generated based on the spatial depth image; the spatial prompt information corresponds to at least one pixel in the spatial depth image; the perceived intensity of the spatial prompt information is associated with the depth information of the corresponding pixel in the spatial depth image; the spatial prompt information is displayed to the user to indicate the environmental conditions in the space where the user is located through the spatial prompt information.

[0202] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0204] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0205] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0207] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0208] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0209] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. A space display method, characterized in that: include: Obtain a spatial depth image of the space where the user is located; The spatial depth image includes depth information corresponding to each pixel point; generating spatial prompt information based on the spatial depth image; The spatial hint information corresponds to at least one pixel in the spatial depth image; The perceived intensity of the spatial cue information is associated with the depth of field information of the corresponding pixel point in the spatial depth image; The space prompt information is displayed to the user, so as to indicate the environment situation in the space where the user is located through the space prompt information.

2. The method according to claim 1, characterized in that After obtaining the spatial depth image of the space where the user is located, the method further includes: The spatial depth image is mapped to a display interface on a user side to display the spatial depth image to the user.

3. The method according to claim 2, characterized in that The generating of spatial prompt information based on the spatial depth image includes: In response to a touch instruction on the display interface, obtaining a touch position of the user on the display interface; Acquiring depth of field information of a pixel point corresponding to the touch position in the spatial depth image according to a touch position of the user in the display interface; Based on the depth of field information corresponding to the touch position, vibration prompt information corresponding to the touch position is generated; wherein the vibration prompt information is used to indicate the distribution of objects in the space where the user is located; the perceived intensity of the vibration prompt information is determined by at least one of the vibration intensity, the vibration frequency, and the vibration waveform; the perceived intensity of the vibration prompt information changes with the change of the depth of field information corresponding to the touch position; The displaying of the space prompt information to the user includes: Based on the perceived intensity of the vibration prompt information, a vibration effect of the vibration prompt information is triggered through a tactile component on the user side; and the vibration effect of the vibration prompt information changes with changes in the perceived intensity.

4. The method according to claim 3, characterized in that The generating vibration prompt information corresponding to the touch position based on the depth of field information corresponding to the touch position includes: According to the depth of field value corresponding to the touch position, the vibration intensity of the vibration prompt information corresponding to the touch position is set; The vibration intensity of the vibration prompt information is inversely proportional to the depth of field value corresponding to the touch position in the spatial depth image.

5. The method according to claim 1, characterized in that After obtaining the spatial depth image of the space where the user is located, the method further includes: Performing image semantic segmentation on the spatial depth image to obtain an object corresponding to each pixel in the spatial depth image; The generating of spatial prompt information based on the spatial depth image includes: According to the objects corresponding to the pixels in the spatial depth image and the corresponding relationship between the objects and the obstacle hazard levels, vibration warning information matching the obstacle hazard level is set for each pixel; The perceived intensity of the vibration prompt information is determined by at least one of vibration intensity, vibration frequency, and vibration waveform.

6. The method according to claim 5, characterized in that The vibration frequency of the vibration prompt information is proportional to the obstacle danger level of the corresponding pixel point in the spatial depth image; and / or The vibration intensity of the vibration prompt information is proportional to the obstacle danger level of the corresponding pixel point in the spatial depth image.

7. The method according to claim 1, characterized in that The generating of spatial prompt information based on the spatial depth image includes: In response to a voice prompt trigger command issued by a user, a target pixel point is selected based on the depth information of each pixel point in the spatial depth image; each target pixel point corresponds to each object in the space where the user is located; and the difference between the depth value of each target pixel point and the depth value of an adjacent pixel point meets a preset condition; Generate voice prompt information corresponding to each target pixel; The perceived intensity of the voice prompt information is determined by at least one of a playback order, a playback volume, and a stereo orientation of a sound source.

8. The method according to claim 7, characterized in that The displaying of the space prompt information to the user includes: According to the depth of field values ​​of each target pixel point in the spatial depth image, the voice prompt information corresponding to each target pixel point is played in sequence at a preset speed; or In response to a scanning instruction issued to the space where the user is located, based on the scanning direction and scanning speed included in the scanning instruction, play voice prompt information corresponding to each target pixel point; Among them, the playback order of voice prompt information corresponding to each target pixel point is inversely proportional to the depth of field value corresponding to each target pixel point; and / or, the playback volume of the voice prompt information corresponding to each target pixel point is inversely proportional to the depth of field value of each target pixel point; and / or, the stereo orientation of the sound source of the voice prompt information corresponding to each target pixel point is consistent with the position direction of each target pixel point mapped to the user's space.

9. A space display device, characterized in that: The device comprises: An acquisition module is used to acquire a spatial depth image of the space where the user is located; the spatial depth image includes depth information corresponding to each pixel point; A generating module, configured to generate spatial hint information based on the spatial depth image; the spatial hint information corresponds to at least one pixel in the spatial depth image; and the perceived intensity of the spatial hint information is associated with the depth of field information of the corresponding pixel in the spatial depth image; The display module is used to display the space prompt information to the user, so as to indicate the environment situation in the space where the user is located through the space prompt information.

10. An electronic device, characterized in that: The electronic device includes a memory and one or more processors; wherein the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the spatial display method as described in any one of claims 1 to 8.

Images