Barrier-free auxiliary control method and device and object searching method and device

Abstract

The invention discloses a barrier-free auxiliary control method and device and an object searching method and device, and belongs to the technical field of electronic equipment. The method comprises the following steps: displaying a scanning interface; the scanning interface is used for displaying a scanning image, and the scanning image is an image obtained by scanning the first space by the image acquisition device; receiving a first input of a scanning completion control in the scanning interface; displaying a spatial three-dimensional image in response to the first input; the spatial three-dimensional image is generated according to the scanned image, the spatial three-dimensional image comprises at least one object in the first space, and the spatial three-dimensional image is used for indicating coordinate information of each object in the first space; in response to the second input, sharing the spatial three-dimensional image to at least one second electronic device; wherein the second electronic equipment outputs a first audio frequency under the condition that the second electronic equipment receives voice input used for searching the first object, the first object is any object in the first space, and the first audio frequency is used for prompting the position of the first object.

Description

Technical Field

[0001] This application belongs to the technical field of electronic devices, and particularly relates to a barrier-free auxiliary control method and device, and an object search method and device. Background Art

[0002] In the related art, in order to provide convenience for people with disabilities, barrier-free service functions are set in related electronic devices such as mobile phones. For example, a visual impairment assistance function is provided. The currently common visual impairment assistance function is closed-loop within the user system, that is, the user needs to first determine an object, and then the mobile phone identifies the type of the object. Visually impaired users often need to perform multiple operations to successfully scan and identify the object, with cumbersome operations and a low success rate of object recognition. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a barrier-free auxiliary control method and device, and an object search method and device, which can solve the problems of cumbersome operations and low success rate of object recognition in the related art for the visual impairment assistance function.

[0004] In a first aspect, the embodiments of this application provide a barrier-free auxiliary control method, which is applied to a first electronic device with a barrier-free auxiliary function. The first electronic device includes an image acquisition device, and the method includes:

[0005] Display a scanning interface; the scanning interface is used to display a scanning image, and the scanning image is an image obtained by the image acquisition device scanning a first space;

[0006] Receive a first input to a scanning completion control in the scanning interface;

[0007] In response to the first input, display a three-dimensional space image; the three-dimensional space image is generated based on the scanning image, and the three-dimensional space image includes at least one object in the first space. The three-dimensional space image is used to indicate the coordinate information of each object in the first space;

[0008] Receive a second input;

[0009] In response to the second input, share the three-dimensional space image to at least one second electronic device; wherein, when the second electronic device receives a voice input for searching for a first object, a first audio is output. The first object is any object in the first space, and the first audio is determined based on the coordinate information of the first object in the first space. The first audio is used to prompt the position of the first object.

[0010] In a second aspect, the embodiments of this application provide an object search method, which is applied to a second electronic device with a barrier-free auxiliary function, and the method includes:

[0011] Receive a three-dimensional space image shared by any first electronic device;

[0012] In the case of receiving a voice input for finding a first object, a first audio is output; wherein, the first audio is determined based on the coordinate information of the first object in the first space, and the first audio is used to prompt the position of the first object.

[0013] In a third aspect, an embodiment of the present application provides a barrier-free auxiliary control device, which includes:

[0014] A display module, configured to display a scanning interface; the scanning interface is used to display a scanning image, and the scanning image is an image obtained by an image acquisition device scanning the first space;

[0015] A receiving module, configured to receive a first input to a scanning completion control in the scanning interface;

[0016] The display module is further configured to, in response to the first input, display a three-dimensional space image; the three-dimensional space image is generated according to the scanning image, and the three-dimensional space image includes at least one object in the first space, and the three-dimensional space image is used to indicate the coordinate information of each object in the first space;

[0017] The receiving module is further configured to receive a second input;

[0018] A sharing module, configured to, in response to the second input, share the three-dimensional space image to at least one second electronic device; wherein, in the case of receiving a voice input for finding a first object, the second electronic device outputs a first audio, the first object is any object in the first space, the first audio is determined based on the coordinate information of the first object in the first space, and the first audio is used to prompt the position of the first object.

[0019] In a fourth aspect, an embodiment of the present application provides an object finding device, which includes:

[0020] A receiving module, configured to receive a three-dimensional space image shared by any first electronic device;

[0021] An output module, configured to, in the case of receiving a voice input for finding a first object, output a first audio; wherein, the first audio is determined based on the coordinate information of the first object in the first space, and the first audio is used to prompt the position of the first object.

[0022] In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the methods in the first aspect and the second aspect are implemented.

[0023] Sixth aspect, embodiments of the present application provide a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the methods in the first aspect and the second aspect are implemented.

[0024] Seventh aspect, embodiments of the present application provide a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the methods in the first aspect and the second aspect.

[0025] Eighth aspect, embodiments of the present application provide a computer program product, which is stored in a storage medium, and the program product is executed by at least one processor to implement the methods in the first aspect and the second aspect.

[0026] In this embodiment, a method for assisting visually impaired users to find items is provided. By using the mobile phones of the family members, friends or other caregivers of the visually impaired users to scan the images of various furniture and items in spaces such as rooms, a three-dimensional space image that can reflect the placement positions of the items is obtained and shared with the device held by the visually impaired user. At this time, the device that receives the three-dimensional space image can accurately identify the accurate positions of each object in the current environment based on the received three-dimensional space image, so as to provide an object search assistance function based on voice recognition for the visually impaired user, simplify the operation of the visually impaired assistance function, and improve the object recognition success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The flowchart of the barrier-free assistance control method according to some embodiments of the present application is shown;

[0028] Figure 2 The schematic diagram of the scanning interface according to some embodiments of the present application is shown;

[0029] Figure 3 The schematic diagram of the preview interface according to some embodiments of the present application is shown;

[0030] Figure 4 The schematic diagram of the barrier-free function interface according to some embodiments of the present application is shown;

[0031] Figure 5 The flowchart of the object search method according to some embodiments of the present application is shown;

[0032] Figure 6 The structural block diagram of the barrier-free assistance control device according to some embodiments of the present application is shown;

[0033] Figure 7 The structural block diagram of the object search device according to some embodiments of the present application is shown;

[0034] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present application is shown;

[0035] Figure 9 It is a schematic diagram of the hardware structure of an electronic device for implementing an embodiment of the present application. Detailed implementation manners

[0036] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0037] The terms "first", "second", etc. in the description and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such terms may be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and the number of objects is not limited. For example, the first object may be one or multiple. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0038] Next, in conjunction with the accompanying drawings, the barrier-free assistance control method and device, and the object search method and device provided in the embodiments of the present application will be described in detail through specific embodiments and their application scenarios.

[0039] In some embodiments of the present application, a barrier-free assistance control method is provided, which is applied to a first electronic device with a barrier-free assistance function. The first electronic device includes an image acquisition device. Figure 1 A flowchart of the barrier-free assistance control method according to some embodiments of the present application is shown, as Figure 1 shown, the method includes:

[0040] Step 102, display a scanning interface; the scanning interface is used to display a scanning image, and the scanning image is an image obtained by the image acquisition device scanning a first space.

[0041] In this embodiment, the first electronic device is a device held by a family member, friend or caregiver of a visually impaired user. Exemplarily, the electronic device includes, but is not limited to, a mobile phone, a tablet computer, a wearable smart device or a dedicated second electronic device. The first electronic device is provided with an image acquisition device. Taking the first electronic device as a mobile phone as an example, the image acquisition device is the camera of the mobile phone.

[0042] Family members, friends or caregivers of visually impaired users can enable the "assistive search" function through the visually impaired assistance function module of the first electronic device, and display the scanning interface. In the scanning interface, the first electronic device calls the image acquisition device to scan the image information of the first space where it is currently located, such as the image information of the living room and bedroom spaces, to obtain the corresponding scanned image.

[0043] Exemplarily, Figure 2 The interface schematic diagram of the scanning interface of some embodiments of the present application is shown. As Figure 2 shown, in the scanning interface 200, the image scanned by the image acquisition device is displayed.

[0044] At this time, the user holding the first electronic device can walk indoors and continuously scan various objects in the room, such as furniture, items, walls, doors and windows, etc. through the image acquisition device. The first electronic device collects the direction information and distance information during the user's scanning process through sensors such as the direction sensor, walking sensor, and attitude sensor, and constructs a three-dimensional space coordinate system through spatial calculation and positioning capabilities. If only a three-dimensional object coordinate information library is generated for a specific room, there is no need to walk in all rooms.

[0045] The barrier-free assistance program can recognize each frame of the image collected by the image acquisition device, use algorithms and artificial intelligence (AI) capabilities to identify the characteristics of each object in the image, identify the item names therein, such as chairs, cups, walls, etc., and combine the direction information and distance information collected by other sensors to construct an information library containing the spatial coordinates of each object.

[0046] Step 104, receive a first input to the scanning completion control in the scanning interface;

[0047] Step 106, in response to the first input, display a three-dimensional space image; the three-dimensional space image is generated based on the scanned image, and the three-dimensional space image includes at least one object in the first space, and the three-dimensional space image is used to indicate the coordinate information of each object in the first space.

[0048] In this embodiment, as Figure 2As shown, a scan completion control 202 is also displayed in the scan interface 200. After the user completes the scan of the current indoor environment and objects, the scan can be completed by clicking the scan completion control 202 button. At this time, the accessibility assistance program will generate a three-dimensional spatial image containing the three-dimensional object coordinate information. In this three-dimensional spatial image, based on the actual scan results, the actual position of each scanned object in the room is displayed. The current user can confirm whether the scan results of the object positions are accurate and whether the item recognition results are accurate by visually viewing the three-dimensional spatial image. When the scan results are inaccurate, the user can restart the scan process.

[0049] Step 108, receive a second input;

[0050] Step 110, in response to the second input, share the three-dimensional spatial image to at least one second electronic device; wherein, when the second electronic device receives a voice input for finding a first object, a first audio is output, the first object is any object in the first space, the first audio is determined based on the coordinate information of the first object in the first space, and the first audio is used to prompt the position of the first object.

[0051] After the scan is completed, the user can share the scanned three-dimensional spatial image, including the coordinate information of each object in the current room, that is, the first space, to the second electronic device held by the visually impaired user. Among them, the second electronic device can be a mobile phone, a tablet computer, a wearable intelligent device or other specialized accessibility assistance products.

[0052] After receiving the three-dimensional spatial image, the second electronic device immediately masters the real coordinate information of each object in the first space. At this time, the visually impaired user can find a specific first object by means of a voice command. For example, asking "Where is the water cup?".

[0053] After receiving the voice input, the second electronic device detects the keyword "water cup" through a voice recognition algorithm. Through the three-dimensional spatial image, the spatial coordinates of the water cup are recognized, and at the same time, combined with the positioning of the current second electronic device, the position of the water cup is determined, and an electronic voice audio, that is, the above-mentioned first audio, is generated through the Text To Speech (TTS) function and output. For example, output "The water cup is in the direction of your right hand, 1 meter away from you".

[0054] Exemplarily, the second electronic device can play the first audio through a speaker set by itself.

[0055] Exemplarily, the second electronic device can communicate with the user's headset through wireless communications such as Bluetooth, Wi-Fi (Wireless Fidelity), etc., and send the first audio to the user's headset for playback.

[0056] In the embodiments of the present application, by using the mobile phones of the family members, friends or other caregivers of visually impaired users to scan the images of various furniture and items in a space such as a room, a three-dimensional space image that can reflect the placement positions of the items is obtained and shared with the device held by the visually impaired user. At this time, the device that receives the three-dimensional space image can accurately identify the accurate positions of each object in the current environment based on the received three-dimensional space image, so as to provide an object search assistance function based on voice recognition for the visually impaired user, simplify the operation of the visually impaired assistance function, and improve the object recognition success rate.

[0057] In some embodiments of the present application, after displaying the three-dimensional space image, the method further includes:

[0058] Receiving a third input for a second object in the three-dimensional space image;

[0059] In response to the third input, controlling to move the display position of the second object in the three-dimensional space image; and updating the coordinate information of the second object in the first space based on the moved display position of the second object.

[0060] In this embodiment, after generating the three-dimensional space image, the user currently performing the scanning operation can perform further editing and modification processing based on the displayed three-dimensional space image, so as to correct possible position deviations.

[0061] Exemplarily, Figure 3 shows a schematic diagram of the preview interface of some embodiments of the present application, as Figure 3 shown, after the user clicks the scan completion control, the preview interface 300 is displayed, and the three-dimensional space image 302 obtained by scanning is in the preview interface 300. In the three-dimensional space image 302, one or more objects are included, including the second object 304.

[0062] The user can judge whether the position of each object is accurate by visually observing the three-dimensional space image 302. When it is necessary to correct the position of an object, such as the second object 304, the user can select and drag the second object 304 through the third input, so as to change the display position of the second object 304 in the three-dimensional image.

[0063] After the display position of the second object 304 is moved, the program automatically updates the coordinate information of the second object 304 in the first space according to the moved display position, so as to ensure the accurate position of the scanned item.

[0064] The present application can enable the user to conveniently correct the position of the scanned object through intuitive and visual operations, thereby improving the accuracy of the scanned three-dimensional space image.

[0065] In some technical solutions of the present application, optionally, before displaying the scanning interface, the method further includes:

[0066] Displaying an accessibility function interface; the accessibility function interface includes a scanning control;

[0067] Receiving a fourth input to the scanning control;

[0068] In response to the fourth input, controlling the image acquisition device to scan the first space and displaying the scanning interface.

[0069] In this embodiment, Figure 4 shows a schematic diagram of the accessibility function interface of some embodiments of the present application. As Figure 4 shown, in the accessibility function interface 400, a scanning control 402 is displayed. After the user clicks the scanning control 402 through the fourth input, the electronic device calls the image acquisition device, such as the rear camera, to start collecting the environmental image of the first space and jumps to display the scanning interface 200 as Figure 2 shown.

[0070] At this time, the accessibility program acquires each frame of image in the scanning interface 200, identifies the objects therein, and combines the direction information and distance information detected by other sensors to generate a three-dimensional space image.

[0071] In the present application, a three-dimensional space image is scanned and generated through an electronic device such as the mobile phone of a family member, friend or caregiver. Therefore, visually impaired users do not need to grope for themselves and scan the objects to be identified, simplifying the operation steps for visually impaired users to use the item search function.

[0072] In some technical solutions of the present application, optionally, after controlling the image acquisition device to scan the first space and displaying the scanning interface, the method further includes:

[0073] In the case of scanning any third object in the first space, displaying a prompt window; the prompt window includes a marking control;

[0074] Receiving a fifth input to the marking control; the fifth input is used to input a reminder message;

[0075] In response to the fifth input, marking the third object as a reminder object and performing an association process on the reminder object and the reminder message; wherein, when the second electronic device receives the three-dimensional space image, and the distance between the current position of the second electronic device and the position of the reminder object is less than the distance threshold, the second electronic device outputs a second audio, and the second audio is determined according to the reminder message.

[0076] In this embodiment, during the scanning process, when encountering important items such as drugs, etc., these items can be marked with emphasis, and important reminder information can be added to the items.

[0077] Exemplarily, during the scanning of the first space, when the third object is scanned, the user can pause and focus the camera screen on the third object. When the accessibility program recognizes the third object, a prompt window is displayed, such as prompting "Do you want to add a reminder?".

[0078] At this time, the user can mark the recognized third object as a reminder object by clicking the marking control in the window and input the corresponding reminder information, such as "There is prepared medicine on the table, remember to take it, there is water in the cup, be careful when taking it", etc. Then, the accessibility program will automatically mark the third object as a reminder object and associate it with the reminder information input by the user.

[0079] After sending the three-dimensional space image to the second electronic device of the visually impaired user, the second electronic device will detect the current position and determine the distance between the current position and the position of each marked reminder object. When it is detected that the visually impaired user is approaching a marked object, such as when it is detected that the distance between the second electronic device and a marked object is less than 1 meter, the second audio generated based on the input reminder information is played. For example, the voice "Remember to take medicine" is played.

[0080] This application can mark important items and automatically play the prompted audio when the visually impaired user approaches the important items, enabling active reminder, thereby providing richer auxiliary functions for the visually impaired user.

[0081] In some embodiments of this application, an object search method is provided, which is applied to a second electronic device with an accessibility assistance function. Figure 5 The flowchart of the object search method according to some embodiments of this application is shown, as Figure 5 shown, the method includes:

[0082] Step 502, receiving a three-dimensional space image shared by any first electronic device;

[0083] Step 504, when receiving a voice input for searching for a first object, outputting a first audio; wherein, the first audio is determined based on the coordinate information of the first object in the first space, and the first audio is used to prompt the position of the first object.

[0084] In this embodiment, the second electronic device may be a mobile phone, a tablet computer, a wearable intelligent device, or other specialized barrier-free assistance products. The second electronic device receives the three-dimensional space image generated by scanning of the first electronic device by family members, friends, or caregivers. The three-dimensional space image carries the first space where the user is currently located, such as the coordinate information of each object in the living room.

[0085] When a visually impaired user needs to find an item, the user can input through voice, such as saying "Where is the water cup". After receiving the voice input, the second electronic device detects the keyword "water cup" through a voice recognition algorithm. Through the three-dimensional space image, the spatial coordinates of the water cup are identified, and at the same time, combined with the positioning of the current second electronic device, the position of the water cup is determined, and an electronic voice audio, that is, the above-mentioned first audio, is generated through the TTS function and output. For example, output "The water cup is in the direction of your right hand, 1 meter away from you".

[0086] In the embodiment of the present application, by receiving the three-dimensional space image sent by the mobile phone of family members, friends, or other caregivers and sharing it with the device held by the visually impaired user. At this time, the device that receives the three-dimensional space image can accurately identify the accurate position of each object in the current environment according to the received three-dimensional space image, so as to provide an object search assistance function based on voice recognition for the visually impaired user, simplify the operation of the visually impaired assistance function, and improve the object recognition success rate.

[0087] In some embodiments of the present application, after receiving the three-dimensional space image shared by any electronic device, the method further includes:

[0088] Obtain the current position of the second electronic device;

[0089] When the distance between the current position of the second electronic device and the position of the reminder object is less than the distance threshold, output a second audio, and the second audio is determined according to the reminder information, and the reminder information is associated with the reminder object.

[0090] In this technical solution, after receiving the three-dimensional space image, when the second electronic device is within the first space corresponding to the three-dimensional space image, the second electronic device continuously detects the current position coordinates, and determines the distance between the user and the reminder object according to the coordinate information of the reminder object in the three-dimensional space image.

[0091] When it is detected that the user is approaching a marked object, such as when it is detected that the distance between the second electronic device and a marked object is less than 1 meter, then play the second audio generated based on the input reminder information. For example, play the voice audio of "Remember to take medicine".

[0092] This application can automatically play a prompt audio when a visually impaired user approaches an important item, enabling active reminder and thus providing richer auxiliary functions for visually impaired users.

[0093] In some embodiments of this application, after outputting the first audio, the method further includes:

[0094] Outputting a third audio; wherein the third audio is used to prompt a first movement path, and the first movement path is a movement path starting from the current position of the second electronic device and ending at the position of the first object.

[0095] In this embodiment, after the second electronic device locates the coordinate information of the object queried by the user, it can guide the user to move near the item queried by the user within the current first space by outputting the third audio. Exemplarily, the second electronic device takes the current position coordinates as the starting point and the position of the first object searched by the user as the ending point to plan the first movement path. This first movement path can automatically avoid obstacles between the user and the first object. The user can come near the first object according to the guidance of the third audio. When the user approaches the first object, a prompt sound is output to inform the user that the first object is nearby.

[0096] For example, when the user asks "Where is the water cup?", the second electronic device identifies the spatial coordinates of the water cup through the three-dimensional spatial image and outputs the first audio "The water cup is in your right hand direction, 1 meter away from you". Then, the second electronic device combines its own coordinate position with the coordinate position of the water cup to plan the first path and generate the third audio. For example, "Please turn 90° to the right, and then take 1 step forward, and the water cup will be in front of you".

[0097] This application can guide the user to move near the object being searched for, helping the user quickly find the item needed, thus providing a more friendly barrier-free service experience for visually impaired users.

[0098] For the barrier-free auxiliary control method provided in the embodiments of this application, the execution subject can be a barrier-free auxiliary control device. In the embodiments of this application, taking the barrier-free auxiliary control device executing the barrier-free auxiliary control method as an example, the barrier-free auxiliary control device provided in the embodiments of this application is described.

[0099] In some embodiments of this application, a barrier-free auxiliary control device is provided. Figure 6 The structural block diagram of the barrier-free auxiliary control device in some embodiments of this application is shown, as Figure 6 shown, the barrier-free auxiliary control device 600 includes:

[0100] A display module 602, configured to display a scanning interface; the scanning interface is used to display a scanning image, and the scanning image is an image obtained by the image acquisition device scanning the first space;

[0101] A receiving module 604, configured to receive a first input to a scanning completion control in a scanning interface;

[0102] The display module 602 is further configured to, in response to the first input, display a three-dimensional space image; the three-dimensional space image is generated based on a scanned image, and the three-dimensional space image includes at least one object in a first space, and the three-dimensional space image is used to indicate coordinate information of each object in the first space;

[0103] The receiving module 604 is further configured to receive a second input;

[0104] A sharing module 606, configured to, in response to the second input, share the three-dimensional space image to at least one second electronic device; wherein, when the second electronic device receives a voice input for finding a first object, a first audio is output, the first object is any object in the first space, the first audio is determined based on the coordinate information of the first object in the first space, and the first audio is used to prompt the position of the first object.

[0105] In an embodiment of the present application, by using a mobile phone of a family member, friend or other caregiver of a visually impaired user to scan images of various furniture and items in a space such as a room, a three-dimensional space image that can reflect the placement positions of the items is obtained and shared with the device held by the visually impaired user. At this time, the device that receives the three-dimensional space image can accurately identify the accurate position of each object in the current environment based on the received three-dimensional space image, so as to provide a voice recognition-based object search assistance function for the visually impaired user, simplify the operation of the visually impaired assistance function, and improve the object recognition success rate.

[0106] In some embodiments of the present application, the receiving module is further configured to receive a third input to a second object in the three-dimensional space image;

[0107] The barrier-free assistance control device further includes:

[0108] A processing module, configured to, in response to the third input, control to move the display position of the second object in the three-dimensional space image; and update the coordinate information of the second object in the first space based on the display position after the second object is moved.

[0109] The present application can enable the user to conveniently correct the position of the scanned object through intuitive and visual operations, thereby improving the accuracy of the scanned three-dimensional space image.

[0110] In some embodiments of the present application, the display module is further configured to display a barrier-free function interface; the barrier-free function interface includes a scanning control;

[0111] The receiving module is further configured to receive a fourth input to the scanning control;

[0112] The barrier-free auxiliary control device further includes:

[0113] A control module, configured to control the image acquisition device to scan a first space and display a scanning interface in response to a fourth input.

[0114] In this application, a three-dimensional image of the space is scanned and generated through an electronic device such as a mobile phone of a family member, a friend, or a caregiver. Therefore, visually impaired users do not need to fumble around by themselves to scan the objects to be recognized, which simplifies the operation steps for visually impaired users to use the item search function.

[0115] In some embodiments of this application, the display module is further configured to display a prompt window when any third object in the first space is scanned; the prompt window includes a marking control;

[0116] A receiving module is further configured to receive a fifth input to the marking control; the fifth input is used to input a reminder message;

[0117] The barrier-free auxiliary control device further includes:

[0118] A marking module, configured to mark the third object as a reminder object in response to the fifth input and perform an association process on the reminder object and the reminder message; wherein, when the second electronic device receives the three-dimensional image of the space, if the distance between the current position of the second electronic device and the position of the reminder object is less than a distance threshold, the second electronic device outputs a second audio, and the second audio is determined according to the reminder message.

[0119] This application can mark important items and automatically play a prompt audio when a visually impaired user approaches an important item, which can achieve active reminder and thus provide richer auxiliary functions for visually impaired users.

[0120] For the object search method of the second electronic device provided in the embodiments of this application, the execution subject may be the object search device of the second electronic device. In the embodiments of this application, the object search device of the second electronic device is taken as an example to execute the object search method of the second electronic device to illustrate the object search device of the second electronic device provided in the embodiments of this application.

[0121] In some embodiments of this application, an object search device is provided. Figure 7 The structural block diagram of the object search device in some embodiments of this application is shown. As Figure 7 shown, the object search device 700 includes:

[0122] A receiving module 702, configured to receive a three-dimensional image of the space shared by any electronic device;

[0123] An output module 704, configured to output a first audio when receiving a voice input for searching for a first object; wherein the first audio is determined based on coordinate information of the first object in a first space, and the first audio is used to prompt the position of the first object.

[0124] In the embodiments of the present application, a three-dimensional spatial image sent by a mobile phone of a family member, a friend or other caregiver is received and shared with a device held by a visually impaired user. At this time, the device that receives the three-dimensional spatial image can accurately identify the accurate position of each object in the current environment according to the received three-dimensional spatial image, so as to provide a voice recognition-based object search assistance function for the visually impaired user, and simplify the operation steps of the visually impaired assistance function.

[0125] In some embodiments of the present application, the object search device further includes:

[0126] An acquisition module, configured to acquire the current position of the object search device;

[0127] The output module is further configured to output a second audio when the distance between the current position of the object search device and the position of the reminder object is less than a distance threshold, where the second audio is determined according to a reminder message, and the reminder message is associated with the reminder object.

[0128] The present application can automatically play a prompted audio when a visually impaired user approaches an important item, and can achieve active reminder, so as to provide a more abundant assistance function for the visually impaired user.

[0129] In some embodiments of the present application, the output module is further configured to output a third audio; wherein the third audio is used to prompt a first movement path, and the first movement path is a movement path starting from the current position of a second electronic device and ending at the position of the first object.

[0130] The present application can guide a user to approach the object being searched for, and help the user quickly find the required item, so as to provide a more friendly barrier-free service experience for the visually impaired user.

[0131] The control device or object search device for barrier-free assistance in the embodiments of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than terminals. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, an in-vehicle electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It may also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0132] The control device or object search device for barrier-free assistance in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0133] The control device or object search device for barrier-free assistance provided by the embodiments of the present application can implement each process implemented by the above method embodiments. To avoid repetition, it will not be elaborated here.

[0134] Optionally, the embodiments of the present application further provide an electronic device. Figure 8 The structural block diagram of the electronic device according to the embodiments of the present application is shown. As Figure 8 shown, the electronic device 800 includes a processor 802, a memory 804, a program or instruction stored on the memory 804 and executable on the processor 802. When the program or instruction is executed by the processor 802, it implements each process of the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0135] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0136] Figure 9 It is a schematic diagram of the hardware structure of an electronic device for implementing the embodiments of the present application.

[0137] The electronic device 900 includes, but is not limited to, components such as a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910.

[0138] Those skilled in the art can understand that the electronic device 900 may further include a power source (such as a battery) for supplying power to each component. The power source can be logically connected to the processor 910 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 9 The structure of the electronic device shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0139] Among them, the display unit 906 is used to display a scanning interface; the scanning interface is used to display a scanned image, and the scanned image is an image obtained by an image acquisition device scanning a first space.

[0140] The user input unit 907 is used to receive a first input to a scanning completion control in the scanning interface.

[0141] The display unit 906 is further used to display a three-dimensional space image in response to the first input; the three-dimensional space image is generated based on the scanned image, and the three-dimensional space image includes at least one object in the first space, and the three-dimensional space image is used to indicate the coordinate information of each object in the first space.

[0142] The user input unit 907 is further used to receive a second input.

[0143] The processor 910 is used to share the three-dimensional space image to at least one second electronic device in response to the second input; wherein, when the second electronic device receives a voice input for searching for a first object, a first audio is output, the first object is any object in the first space, the first audio is determined based on the coordinate information of the first object in the first space, and the first audio is used to prompt the position of the first object.

[0144] In the embodiment of the present application, by using the mobile phones of the family members, friends or other caregivers of visually impaired users to scan the images of various furniture and items in a space such as a room, a three-dimensional space image that can reflect the placement positions of the items is obtained and shared with the device held by the visually impaired user. At this time, the device that receives the three-dimensional space image can accurately identify the accurate positions of each object in the current environment based on the received three-dimensional space image, so as to provide a function of assisting in finding objects based on voice recognition for visually impaired users, simplifying the operation of the visually impaired assistance function and improving the success rate of object recognition.

[0145] Optionally, the user input unit 907 is further configured to receive a third input for a second object in the three-dimensional spatial image;

[0146] The processor 910 is further configured to, in response to the third input, move the display position of the second object in the three-dimensional spatial image; and update the coordinate information of the second object in the first space based on the display position of the second object after movement.

[0147] This application can enable the user to conveniently correct the position of the scanned object through intuitive and visual operations, thereby improving the accuracy of the three-dimensional spatial image obtained by scanning.

[0148] Optionally, the display unit 906 is further configured to display an accessibility function interface; the accessibility function interface includes a scanning control;

[0149] The user input unit 907 is further configured to receive a fourth input for the scanning control;

[0150] The processor 910 is further configured to, in response to the fourth input, control the image acquisition device to scan the first space and display a scanning interface.

[0151] This application scans and generates a three-dimensional spatial image through an electronic device such as a mobile phone of a family member, friend, or caregiver. Therefore, visually impaired users do not need to explore on their own and scan the objects to be recognized, simplifying the operation steps for visually impaired users to use the item search function.

[0152] Optionally, the display unit 906 is further configured to display a prompt window when any third object in the first space is scanned; the prompt window includes a marking control;

[0153] The user input unit 907 is further configured to receive a fifth input for the marking control; the fifth input is used to input a reminder message;

[0154] The processor 910 is further configured to, in response to the fifth input, mark the third object as a reminder object and perform an association process on the reminder object and the reminder message; wherein, when the second electronic device receives the three-dimensional spatial image and the distance between the current position of the second electronic device and the position of the reminder object is less than a distance threshold, the second electronic device outputs a second audio, and the second audio is determined according to the reminder message.

[0155] This application can mark important items and automatically play a prompt audio when a visually impaired user approaches an important item, enabling active reminder and thus providing richer auxiliary functions for visually impaired users.

[0156] Optionally, the processor 910 is further configured to receive a three-dimensional spatial image shared by any electronic device; and output a first audio in response to receiving a voice input for finding a first object, where the first audio is determined based on the coordinate information of the first object in a first space, and the first audio is used to prompt the position of the first object.

[0157] In the embodiment of the present application, a three-dimensional spatial image sent by the mobile phone of a family member, a friend or other caregiver is received and shared with the device held by the visually impaired user. At this time, the device that receives the three-dimensional spatial image can accurately identify the accurate position of each object in the current environment according to the received three-dimensional spatial image, so as to provide the visually impaired user with an object search assistance function based on voice recognition, and simplify the operation steps of the visually impaired assistance function.

[0158] Optionally, the processor 910 is further configured to obtain the current position of a second electronic device; and output a second audio when the distance between the current position of the second electronic device and the position of a reminder object is less than a distance threshold, where the second audio is determined according to reminder information associated with the reminder object.

[0159] The present application can automatically play a prompted audio when the visually impaired user approaches an important item, enabling active reminder, so as to provide the visually impaired user with a more comprehensive assistance function.

[0160] Optionally, the processor 910 is further configured to output a third audio, where the third audio is used to prompt a first movement path, and the first movement path is a movement path starting from the current position of the second electronic device and ending at the position of the first object.

[0161] The present application can guide the user to approach the object being searched for, helping the user quickly find the item needed, so as to provide a more user-friendly barrier-free service experience for the visually impaired user.

[0162] It should be understood that in the embodiments of the present application, the input unit 904 may include a Graphics Processing Unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes the image data of static pictures or videos obtained by an image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also referred to as a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. The other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0163] The memory 909 can be used to store software programs and various data. The memory 909 mainly includes a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area can store an operating system, applications or instructions required for at least one function (such as a sound output function, an image output function, etc.). In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory may be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 909 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memories.

[0164] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor may not be integrated into the processor 910 either.

[0165] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0166] Among them, the processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disc, etc.

[0167] The embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run a program or instruction to implement each process of the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0168] It should be understood that the chip mentioned in the embodiment of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0169] The embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement each process of the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0170] It should be noted that, in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising such element. In addition, it should be pointed out that the scope of the methods and apparatuses in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0171] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present application.

[0172] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Those of ordinary skill in the art, under the inspiration of the present application and without departing from the purpose of the present application and the scope protected by the claims, can still make many forms, all of which fall within the protection scope of the present application.

Claims

1. A barrier-free auxiliary control method, characterized in that: Applied to a first electronic device with barrier-free auxiliary function, the first electronic device includes an image acquisition device, and the method includes: Display a scanning interface; the scanning interface is used to display a scanned image, the scanned image is an image obtained by the image acquisition device scanning the first space; Receiving a first input to a scan completion control in the scanning interface; In response to the first input, display a spatial three-dimensional image; the spatial three-dimensional image is generated according to the scanned image, the spatial three-dimensional image includes at least one object in the first space, and the spatial three-dimensional image is used to indicate coordinate information of each of the objects in the first space; receiving a second input; In response to the second input, the spatial three-dimensional image is shared to at least one second electronic device; wherein, when the second electronic device receives a voice input for searching for a first object, it outputs a first audio, the first object being any object in the first space, the first audio being determined based on the coordinate information of the first object in the first space, and the first audio being used to prompt the position of the first object.

2. The barrier-free auxiliary control method according to claim 1, characterized in that: After displaying the spatial three-dimensional image, the method further includes: receiving a third input of a second object in the spatial three-dimensional image; In response to the third input, controlling movement of a display position of the second object in the spatial three-dimensional image; and Based on the displayed position of the second object after the movement, the coordinate information of the second object in the first space is updated.

3. The barrier-free auxiliary control method according to claim 1, characterized in that: Before displaying the scanning interface, the method further includes: Displaying an accessibility interface; the accessibility interface including a scanning control; receiving a fourth input to the scan control; In response to the fourth input, the image acquisition device is controlled to scan the first space and the scanning interface is displayed.

4. The barrier-free auxiliary control method according to claim 3, characterized in that: After controlling the image acquisition device to scan the first space and displaying the scanning interface, the method further includes: When any third object in the first space is scanned, a prompt window is displayed; the prompt window includes a marking control; receiving a fifth input to the marking control; the fifth input is used to input reminder information; In response to the fifth input, the third object is marked as a reminder object, and the reminder object and the reminder information are associated with each other; wherein, after the second electronic device receives the spatial three-dimensional image, when the distance between the current position of the second electronic device and the position of the reminder object is less than a distance threshold, the second electronic device outputs a second audio, and the second audio is determined based on the reminder information.

5. An object search method, applied to a second electronic device with barrier-free auxiliary function, characterized in that: The method comprises: Receiving a spatial three-dimensional image shared by any first electronic device; When a voice input for searching for a first object is received, a first audio is output; wherein the first audio is determined based on coordinate information of the first object in a first space, and the first audio is used to indicate the position of the first object.

6. The object search method according to claim 5, characterized in that: After receiving the spatial three-dimensional image shared by any first electronic device, the method further includes: Acquire the current location of the second electronic device; When the distance between the current position of the second electronic device and the position of the reminder object is less than a distance threshold, a second audio is output, where the second audio is determined based on reminder information, and the reminder information is associated with the reminder object.

7. The object search method according to claim 5, characterized in that: After outputting the first audio, the method further includes: Output a third audio; wherein the third audio is used to prompt a first moving path, wherein the first moving path is a moving path starting from the current position of the second electronic device and ending at the position of the first object.

8. A barrier-free auxiliary control device, characterized in that: The barrier-free auxiliary control device comprises: A display module, used to display a scanning interface; the scanning interface is used to display a scanned image, and the scanned image is an image obtained by the image acquisition device scanning the first space; A receiving module, configured to receive a first input to a scan completion control in the scanning interface; The display module is further used to display a spatial three-dimensional image in response to the first input; the spatial three-dimensional image is generated according to the scanned image, the spatial three-dimensional image includes at least one object in the first space, and the spatial three-dimensional image is used to indicate the coordinate information of each of the objects in the first space; The receiving module is further used to receive a second input; A sharing module is used to share the spatial three-dimensional image to at least one second electronic device in response to the second input; wherein, when the second electronic device receives a voice input for searching for a first object, it outputs a first audio, the first object is any object in the first space, the first audio is determined based on the coordinate information of the first object in the first space, and the first audio is used to prompt the position of the first object.

9. The barrier-free auxiliary control device according to claim 8, characterized in that: The receiving module is further used to receive a third input of a second object in the spatial three-dimensional image; The barrier-free auxiliary control device further includes: a processing module, configured to control movement of a display position of the second object in the spatial three-dimensional image in response to the third input; and Based on the displayed position of the second object after the movement, the coordinate information of the second object in the first space is updated.

10. The barrier-free auxiliary control device according to claim 8, characterized in that: The display module is further used to display an accessibility interface; the accessibility interface includes a scanning control; The receiving module is further used to receive a fourth input to the scanning control; The barrier-free auxiliary control device further includes: A control module is used to control the image acquisition device to scan the first space and display the scanning interface in response to the fourth input.

11. The barrier-free auxiliary control device according to claim 10, characterized in that: The display module is further configured to display a prompt window when any third object in the first space is scanned; the prompt window includes a marking control; The receiving module is further used to receive a fifth input to the marking control; The fifth input is used to input reminder information; The barrier-free auxiliary control device further includes: A marking module is used to mark the third object as a reminder object in response to the fifth input, and to associate the reminder object with the reminder information; wherein, after the second electronic device receives the spatial three-dimensional image, when the distance between the current position of the second electronic device and the position of the reminder object is less than a distance threshold, the second electronic device outputs a second audio, and the second audio is determined based on the reminder information.

12. An object search device, characterized in that: The object finding device comprises: A receiving module, configured to receive a spatial three-dimensional image shared by any first electronic device; An output module is used to output a first audio when receiving a voice input for searching for a first object; wherein the first audio is determined based on the coordinate information of the first object in a first space, and the first audio is used to prompt the position of the first object.

13. The object searching device according to claim 12, characterized in that: The object finding device also includes: An acquisition module, used to acquire the current position of the object search device; The output module is further configured to output a second audio when the distance between the current position of the object search device and the position of the reminder object is less than a distance threshold, wherein the second audio is determined according to reminder information associated with the reminder object.

14. The object searching device according to claim 12, characterized in that: The output module is further used to output a third audio; wherein the third audio is used to prompt a first moving path, wherein the first moving path is a moving path starting from the current position of the object searching device and ending at the position of the first object.

Images