A visual assistance method, system and device

Abstract

The embodiment of the present application provides a visual assistance method, system and device, which includes: collecting image data of the environment; performing core calculations based on the image data; the core calculations include any one or more of the following: distance detection, object detection, location detection and scene description generation; determining auxiliary information for the assisted person based on the calculation results of the core calculations; converting the auxiliary information into interactive sentences, and transmitting the interactive sentences to the assisted person in the form of voice. Through this embodiment, the safety factor, sense of security and cognition of the space in which the user is located during travel are improved.

Description

Technical Field

[0001] The present invention relates to visual assistance technology, and more particularly to a visual assistance method, system and device. Background Art

[0002] Artificial intelligence is a technology that uses the powerful computing capabilities of modern computers to learn from a large amount of data, allowing computers to complete tasks like humans. Deep learning uses deep neural networks, that is, many neurons that contain linear transformations and simple nonlinear transformations, and uses a large amount of data and corresponding labels to adjust the parameters of linear transformations through back propagation, so that computers can learn the rules between data and their labels. This technology has important applications in computer vision, natural language processing and other fields.

[0003] Edge computing is a new computing method in the 5G era, and has important applications in facilities such as the Internet of Things. It includes terminals that generate data and receive computing results, edge servers that are close to the terminals and are responsible for most of the computing, and cloud servers that are far away from the terminals and exchange data with edge servers. Its main data flow is to send large amounts of data from the terminal to the edge server, and the edge server will transmit relatively less data to the cloud server after computing. The computing results are returned to the edge server and then to the terminal. It reduces the long-distance transmission of large amounts of data, which is very consistent with the underlying characteristics of 5G technology.

[0004] Modern visual assistance technologies mostly use radar, binocular cameras and other input hardware, which are relatively expensive and have few ways to purchase them. In addition, the few systems that use artificial intelligence technology mostly use single-chip microcomputers for calculations, which are extremely inefficient. Most visual assistance technologies have not conducted in-depth interactive research, and only report cold data to users through voice, which increases the user's mental workload and reduces the real-time and simplicity of the technology. These devices cannot be popularized on a large scale, resulting in the main auxiliary tool for most blind people to travel is still the blind stick. The blind stick is inefficient and requires people to actively touch the ground to explore, causing many inconveniences in travel. Summary of the invention

[0005] The embodiments of the present application provide a visual assistance method, system and device that can improve the user's safety factor, sense of security and understanding of the space they are in during travel.

[0006] The present application embodiment provides a visual assistance method, which may include:

[0007] Collect image data of the environment;

[0008] Performing core calculations based on the image data; the core calculations include any one or more of the following: distance detection, object detection, location detection, and scene description generation;

[0009] Determining auxiliary information for the assisted person according to the calculation result of the core calculation;

[0010] The auxiliary information is converted into interactive sentences, and the interactive sentences are transmitted to the assisted person in the form of voice.

[0011] In an exemplary embodiment of the present application, the image data may include: pictures and / or videos; and performing core calculations according to the image data may include:

[0012] Inputting the collected picture into a pre-trained first neural network, determining a feature object in the picture through the first neural network, and measuring the depth between the feature object and a shooting device of the picture to achieve the distance detection;

[0013] Inputting the collected picture into a pre-trained second neural network, detecting whether there is an object and the type of the object in the picture through the second neural network, and measuring the position of the object to achieve the object detection; the position of the object includes the orientation of the object relative to the person being assisted;

[0014] Inputting the collected picture into a pre-trained third neural network, detecting the scene in the picture through the third neural network, and predicting n possible scenes, where n is a positive integer, to achieve the location detection;

[0015] The captured video is input into a pre-trained fourth neural network, and voice description information of events in the video is generated by the fourth neural network to achieve the scene description generation.

[0016] In an exemplary embodiment of the present application, the first neural network may include a SeNet-154 neural network, and use a NYU-v2 dataset as a training dataset;

[0017] The second neural network may include a Darkent-v3 neural network and implement object detection based on a yolo-v3 algorithm;

[0018] The third neural network may include a ResNet-152 neural network, and the Place365 data set is used as a training data set;

[0019] The fourth neural network may include: an encoder ResNet-152 and a decoder SA-LSTM.

[0020] In an exemplary embodiment of the present application, the method may further include: the image data may include a picture; before performing core calculation according to the image data, preprocessing the picture;

[0021] The preprocessing of the image may include:

[0022] Screen out qualified images from all collected images and remove unqualified images; and,

[0023] The qualified image is divided into multiple regions, wherein adjacent regions have overlapping parts, so that it can adapt to the input of the neural network and improve the recognition accuracy.

[0024] The screening of unqualified images includes using computer vision algorithms to calculate image entropy and perform blur detection, and using device posture data to screen blurry images.

[0025] In an exemplary embodiment of the present application, the auxiliary information may include any one or more of the following: distance warning, object prompt, location change prompt, scene description, and active guidance;

[0026] Wherein, the distance alarm is used to issue a reminder to the assisted person when the distance between the assisted person and the detected object is less than or equal to a preset distance threshold;

[0027] The object prompt is used to describe the object information corresponding to the detected object to the assisted person; the object information may include: name and / or spatial coordinates;

[0028] The location change prompt is used to prompt the assisted person that the current location has changed and provide the assisted person with multiple possible locations;

[0029] The scene description is used to describe the dynamic scene in the captured video in words;

[0030] The active guidance is used to determine obstacles based on the object depth information obtained in the distance detection, and to provide road direction guidance to the assisted person in the form of voice.

[0031] In an exemplary embodiment of the present application, determining the assistance information for the assisted person according to the calculation result of the core calculation may include:

[0032] Determining the depth information detected by the distance detection as a first distance between an arbitrary first object on the picture and the assisted person, comparing the first distance with the distance threshold, and generating the distance alarm when the first distance is less than or equal to the distance threshold;

[0033] Determine a first name of the first object according to the type of object detected in the object detection; determine a first height of the first object according to the object boundary information detected in the object detection; determine whether the first object can be suspended in the air according to the first name; determine a first spatial coordinate of the first object according to the first distance, the first height, and a first position of the first object relative to the assisted person detected in the object detection; correct the first spatial coordinate of the first object according to the information of whether the first object can be suspended in the air; generate a prompt content of the object prompt according to the first name, the corrected first spatial coordinate, and the suspended state of the first object;

[0034] Retrieving n possible locations predicted last time according to the location detection, and comparing first location information obtained during the current location detection process with second location information of the n possible locations predicted last time, and when a difference rate between the first location information and the second location information is greater than or equal to a preset difference rate threshold, determining that the current location has changed, and re-predicting n possible locations according to the first location information; generating prompt content of the location change prompt according to the information of the change in the current location and the re-predicted n possible locations;

[0035] Retrieving the voice description information obtained in the scene description generation calculation, and using the voice description information as the description content of the scene description;

[0036] According to the depth information detected in the distance detection, it is determined whether there is a continuous segment around the assisted person whose depth is greater than or equal to a preset depth threshold. When the continuous segment exists, the continuous segment is determined as a feasible route as the guidance direction of the active guidance.

[0037] In an exemplary embodiment of the present application, the coordinate system where the spatial coordinates are located may take the center of gravity of the assisted person as the coordinate origin, and the coordinate system may include: a depth coordinate, a horizontal coordinate, and a vertical coordinate;

[0038] The direction in which the assisted person is facing in the horizontal direction is the positive direction of the depth coordinate, the direction in which the left hand side of the assisted person is located in the horizontal direction is the positive direction of the horizontal coordinate, and the vertically upward direction in the vertical direction is the positive direction of the vertical coordinate;

[0039] The horizontal axis is divided into segments a, the vertical axis is divided into segments b, and the vertical axis is divided into segments c, each segment having a corresponding natural language name;

[0040] On the horizontal plane of the coordinate system, the area 180 degrees in front of the person being assisted is divided into d sectors, each sector has a corresponding natural language name; a, b, c, d are positive integers.

[0041] In an exemplary embodiment of the present application, generating the prompt content of the object prompt according to the first spatial coordinates may include:

[0042] When the first spatial coordinate is located on the horizontal coordinate, the vertical coordinate or the ordinate, forming a description content about the first spatial coordinate in the prompt content according to the natural language name corresponding to the segment where the first spatial coordinate is located;

[0043] When the first spatial coordinate changes, the description content of the first spatial coordinate is updated in real time according to the change of the first spatial coordinate.

[0044] In an exemplary embodiment of the present application, the method may further include: determining the auxiliary information to be used according to the current visual assistance mode;

[0045] The visual assistance mode may include any one or more of the following: walking mode, exploration mode, close-up mode and standby mode.

[0046] In an exemplary embodiment of the present application, determining the auxiliary information to be used according to the current visual assistance mode may include:

[0047] When the user is currently in the walking mode, the user is guided in the road direction according to the active guidance, and objects that need to be reminded detected by the object detection are screened out, and the objects that need to be reminded are described according to the object prompt;

[0048] When the exploration mode is currently in progress, a spatial overall description is performed for the currently collected picture at a preset time interval; wherein the picture is divided into a plurality of grids, and the description content includes any one or more of the following: a scene description of each grid, a spatial coordinate description of each grid, a description of the average distance between each grid and the assisted person, and a description of the object information of the objects contained in each grid;

[0049] When the close-up mode is currently in place, the object prompt is executed, wherein when the number of objects of the same type detected is greater than or equal to a preset number threshold, the detected objects of the same type are described together in the prompt content of the object prompt, and when the number of objects of the same type detected is less than the number threshold, the detected objects of the same type are described separately in the prompt content of the object prompt; and the prompt content only includes a description of the horizontal coordinate and / or the vertical coordinate, and does not include a description of the depth coordinate;

[0050] When the current mode is in the standby mode, only the current visual assistance mode is uploaded as the standby mode, so that the calculation is stopped according to the standby mode.

[0051] In an exemplary embodiment of the present application, the method may further include:

[0052] Collecting the input assistance help instruction, and determining the assistance information for the person being assisted according to the assistance help instruction;

[0053] The auxiliary help instruction is a signal used to indicate that the person being assisted needs help; the auxiliary information corresponding to the auxiliary help instruction includes: sending help information.

[0054] The embodiment of the present application also provides a visual assistance system, which may include: an image data acquisition module, a core calculation module, an auxiliary information determination module, and a speech processing module;

[0055] The image data acquisition module is configured to acquire image data of the environment;

[0056] The core computing subsystem is configured to perform core computing based on the image data; the core computing includes any one or more of the following: distance detection, object detection, location detection, and scene description generation;

[0057] The auxiliary information determination module is configured to determine auxiliary information for the assisted person according to the calculation result of the core calculation;

[0058] The speech processing module is configured to convert the auxiliary information into interactive sentences and transmit the interactive sentences to the assisted person in the form of speech.

[0059] In an exemplary embodiment of the present application, the core computing subsystem may include: a distance detection module, an object detection module, a location detection module, and a scene description generation module; the speech processing module includes a speech synthesis module and a speech interaction module;

[0060] Among them, the distance detection module, the object detection module and the location detection module are set on the cloud server, the scene description generation module, the auxiliary information determination module and the speech synthesis module are set on the edge server, and the speech interaction module is set on the mobile terminal.

[0061] In an exemplary embodiment of the present application, the visual assistance system may further include: an input acquisition module and a help-seeking module;

[0062] The input collection module may be configured to collect input assistance help instructions; the assistance help instructions are signals used to indicate that the person being assisted needs help;

[0063] The assistance module may be configured to determine assistance information for the person being assisted according to the assistance assistance instruction; the assistance information corresponding to the assistance assistance instruction includes: sending assistance information.

[0064] An embodiment of the present application also provides a visual assistance device, which may include a processor and a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed by the processor, any one of the visual assistance methods described above is implemented.

[0065] The embodiment of the present application may include: collecting image data of the environment; performing core calculations based on the image data; the core calculations include any one or more of the following: distance detection, object detection, location detection, and scene description generation; determining auxiliary information for the assisted person based on the calculation results of the core calculations; converting the auxiliary information into interactive sentences, and delivering the interactive sentences to the assisted person in the form of voice. Through this embodiment, the safety factor, sense of security, and cognition of the space in which the user is located during travel are improved.

[0066] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0068] Figure 1 This is a flow chart of the visual assistance method according to an embodiment of the present application;

[0069] Figure 2 This is a schematic diagram of module deployment in an embodiment of the present application;

[0070] Figure 3 A schematic diagram of the data exchange process of each module in an embodiment of the present application;

[0071] Figure 4 A schematic diagram of segmented coordinates of an embodiment of the present application;

[0072] Figure 5 A block diagram of the visual assistance system of an embodiment of the present application;

[0073] Figure 6 This is a block diagram of the visual aid device according to an embodiment of the present application. DETAILED DESCRIPTION

[0074] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0075] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

[0076] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0077] The present application embodiment provides a visual assistance method, such as Figure 1 As shown, steps S101-S104 may be included:

[0078] S101, collecting image data of the environment;

[0079] S102, performing core calculation according to the image data; the core calculation includes any one or more of the following: distance detection, object detection, location detection and scene description generation;

[0080] S103, determining the auxiliary information for the assisted person according to the calculation result of the core calculation;

[0081] S104: convert the auxiliary information into interactive sentences, and transmit the interactive sentences to the assisted person in the form of voice.

[0082] In an exemplary embodiment of the present application, a visual assistance system for visually impaired people is provided, which does not require special and complex hardware and has a fast computing speed. It can use artificial intelligence for image processing and use edge computing technology for computing acceleration.

[0083] In the exemplary embodiments of the present application, more attention is paid to the interaction with the user, and the processed information is output in a way that is more in line with the cognitive laws and can be accepted and understood by the user more quickly until consciousness is formed, thereby improving the user's safety factor, sense of security and cognition of the space they are in during travel.

[0084] In the exemplary embodiments of the present application, the embodiments of the present application can be implemented through a common mobile terminal, which includes but is not limited to a smart phone, a tablet computer, a smart wearable device, etc.

[0085] In the exemplary embodiments of the present application, there is no limitation on the operating system used by the mobile terminal. For example, the mobile terminal can be a smart phone equipped with an Android operating system, which has the most basic rear camera, volume up and down buttons, a gyroscope, and an acceleration sensor.

[0086] In the exemplary embodiments of the present application, there is no limitation on the communication technologies supported by the mobile terminal. For example, the communication technologies may include but are not limited to 4G, 5G, 6G, etc.

[0087] In an exemplary embodiment of the present application, software including the embodiment of the present application can be deployed on a mobile terminal, and the software functions may include: taking photos at a frame rate of 30 frames per second and uploading them to the edge server at the same time; accepting user input to the volume key to start or shut down the visual assistance function of the embodiment of the present application, and switching the system mode (visual assistance mode); accepting user input to the volume key, and pressing the volume key at the same time to start or shut down the auxiliary help function of the embodiment of the present application; receiving the speech synthesis result sent back by the edge server, and playing the speech synthesis result to the user (i.e., the person being assisted); receiving the auxiliary help result sent back by the edge server, and playing the auxiliary help result to people around the user.

[0088] In an exemplary embodiment of the present application, Figure 2As shown, the preprocessing module 1, the scene description generation module 2, the surrounding calculation module 4 (which can be called the auxiliary information determination module, realizing the functions of distance alarm, object prompt, location change prompt, scene description, active guidance, etc.) and the speech processing module (which can include the speech synthesis module 51 and the speech interaction module 52) can be deployed on the edge server A, and the monocular ranging module 6, the object detection module 7 and the location detection module 8 can be deployed on the cloud server B. The speech interaction module 52 can be set on the mobile terminal C, for example, it can be a wired headset or a Bluetooth headset; the camera device 9 (such as a camera), the acceleration sensor 10, the gyroscope 11, the vibration module 12 and other devices can be set on the mobile terminal C, and the visual assistance mode 3 can be set. The modules on the edge server A and the cloud server B cooperate with the mobile terminal C to implement the embodiment scheme of the present application. The specific data exchange process of each module can be as follows Figure 3 shown.

[0089] In an exemplary embodiment of the present application, the monocular ranging module, object detection module, and place detection module in the core computing subsystem require less traffic because the time interval between input images is relatively large, and they can be deployed on a cloud server for computing.

[0090] In an exemplary embodiment of the present application, the scene description generation module in the core computing subsystem can be deployed on an edge server for computing because the time interval between input images is relatively small and the required traffic is large.

[0091] In an exemplary embodiment of the present application, since the peripheral computing module requires the calculation results of the core computing subsystem and the amount of calculation is relatively small, it can be deployed on the edge server for calculation.

[0092] In an exemplary embodiment of the present application, the speech synthesis module in the speech processing module may be deployed on an edge server for calculation because the transmitted voice traffic is large and the calculation results of surrounding computing modules are required.

[0093] In an exemplary embodiment of the present application, inputs from a camera, an acceleration sensor, a gyroscope, etc. may be acquired automatically by a software system on a mobile terminal without requiring active operation by a user.

[0094] In an exemplary embodiment of the present application, based on the above deployment, the detailed steps of the visual assistance method for the visually impaired (i.e., the assisted person) based on artificial intelligence and edge computing may include steps 1-7:

[0095] 1. Turn on the camera on the mobile terminal and collect image data at a certain frame rate per second; the image data may include pictures and / or videos. While collecting image data, the posture data of the mobile terminal may be collected, and the posture data may include acceleration data, gyroscope data, etc.

[0096] 2. Through the pre-processing module, the pictures collected by the camera device can be screened, wherein qualified pictures can be called Class 1 pictures, and unqualified pictures can be called Class 2 pictures.

[0097] 3. Send the qualified images to the core computing subsystem on the cloud server (which may include a monocular ranging module, an object detection module, a location detection module, and a scene description generation module) to perform ranging (i.e., distance detection), object detection, location detection, and scene description generation. The posture data of the mobile terminal can be sent to the core computing subsystem together with the qualified images for calculation.

[0098] 4. Send the calculation results to the surrounding calculation module and read the user's mode configuration (i.e. the current visual assistance mode).

[0099] 5. According to the user's mode configuration, a series of peripheral calculations are performed (i.e., the auxiliary information for the assisted person is determined) to convert the data of the core calculation into interactive statements that conform to cognition.

[0100] 6. The interactive sentences are transmitted to the speech synthesis module to synthesize a slow and gentle human voice.

[0101] 7. Play the synthesized human voice to the user through the voice interaction module.

[0102] In an exemplary embodiment of the present application, the method may further include: the image data may include a picture; before performing core calculation according to the image data, preprocessing the picture;

[0103] The preprocessing of the image may include but is not limited to:

[0104] Screen out qualified images from all collected images and remove unqualified images; and,

[0105] The qualified image is divided into a plurality of regions, wherein adjacent regions have overlapping parts.

[0106] In an exemplary embodiment of the present application, the pictures acquired by the camera device may be further screened by using information such as a gyroscope, an acceleration sensor, and the number of color blocks in the picture in the mobile terminal.

[0107] In an exemplary embodiment of the present application, for example, blur caused by the camera moving too fast can be detected, and all blurred pictures are marked as unqualified pictures; blur caused by the camera focusing failure can be detected, and all blurred pictures are marked as unqualified pictures; too little information caused by the camera being too close to a certain plane or object can also be detected, and all pictures with too little information are marked as unqualified pictures.

[0108] In an exemplary embodiment of the present application, after the image is uploaded to the edge server, the detailed steps of preprocessing the image by the preprocessing module may include steps 21-29:

[0109] 21. Use the downsampling method to change the image size to a preset size, such as 224×448.

[0110] 22. Using the dynamic blur detection algorithm in computer vision, the dynamic blurred images are screened out and marked as Class 2 images, i.e., unqualified images.

[0111] 23. Using the gyroscope and acceleration sensor data uploaded from the mobile terminal, the images with dynamic blur are screened out and marked as Class 2 images, i.e., unqualified images.

[0112] 24. Use the Laplace variance algorithm in computer vision to filter out Gaussian blurred images and mark them as Class 2 images, i.e., unqualified images.

[0113] 25. Use the clustering algorithm to cluster pixels with similar distance and color, and detect how many clusters appear in the image. If the number of clusters is too small, it is considered that the image has too little available information and is marked as a Class 2 image, that is, an unqualified image.

[0114] 26. According to the timeline, take out one remaining picture every set time (such as 5 seconds), mark it as Category 1 picture (i.e. qualified picture), and mark the rest as Category 2 pictures.

[0115] 27. Image segmentation: segment all images of category 1 into upper and lower parts, and then combine the lower half of the upper half with the upper half of the lower half to form the middle part. For example, the vertical coordinates of the image of category 1 are divided into two segments, 224×[1,224] for the upper half of the image, and 224×[225,448] for the lower half of the image. Generate the middle half of the image, which is 224×[112,335].

[0116] 28. Image screening: extract one image from all category 1 images every certain number of frames and mark them as selected.

[0117] 29. Send the category 1 images, the segmentation results of category 1 images, and category 2 images to the cloud server in chronological order.

[0118] In an exemplary embodiment of the present application, the cloud server can simultaneously start the three modules deployed therein (i.e., the monocular ranging module, object detection module, place detection module and scene description generation module included in the core computing subsystem) to perform parallel core computing.

[0119] In an exemplary embodiment of the present application, the image data may include: pictures and / or videos; and performing core calculations according to the image data may include:

[0120] Inputting the collected picture into a pre-trained first neural network, determining a feature object in the picture through the first neural network, and measuring the depth between the feature object and a shooting device of the picture to achieve the distance detection;

[0121] Inputting the collected picture into a pre-trained second neural network, detecting whether there is an object and the type of the object in the picture through the second neural network, and measuring the position of the object to achieve the object detection; the position of the object includes the orientation of the object relative to the person being assisted;

[0122] Inputting the collected picture into a pre-trained third neural network, detecting the scene in the picture through the third neural network, and predicting n possible scenes, where n is a positive integer, to achieve the location detection;

[0123] The captured video is input into a pre-trained fourth neural network, and voice description information of events in the video is generated by the fourth neural network to achieve the scene description generation.

[0124] In an exemplary embodiment of the present application, the first neural network may include a SeNet-154 neural network, and use a NYU-v2 dataset as a training dataset;

[0125] The second neural network may include a Darknet-v3 neural network, which implements object detection based on the yolo-v3 algorithm;

[0126] The third neural network may include a ResNet-152 neural network, and the Place365 data set is used as a training data set;

[0127] The fourth neural network may include: an encoder ResNet-152 and a decoder SA-LSTM.

[0128] In an exemplary embodiment of the present application, the monocular ranging module includes the first neural network, the object detection module includes the second neural network, the place detection module includes the third neural network, and the scene description generation module includes the fourth neural network.

[0129] In an exemplary embodiment of the present application, the monocular ranging module uses image input from a camera device to estimate the distance. It does not require a complex binocular camera in traditional methods, nor does it require camera calibration. An ordinary smartphone camera can serve as the input end of this monocular ranging module.

[0130] In an exemplary embodiment of the present application, the monocular ranging module can utilize artificial intelligence algorithms for calculation, use the latest SeNet-154 neural network, and utilize encoders, decoders, feature fusers of different sizes, and refiners for depth detection.

[0131] In an exemplary embodiment of the present application, the monocular ranging module adopts SeNet-154 as the backbone neural network, which is also an encoder. It can be divided into 4 blocks and a single convolution layer according to the SE module. The outputs of the last three blocks and the convolution layer are input into the multi-scale feature fuser, the output of the single convolution layer is input into the decoder, and the outputs of the decoder and the feature fuser are input into the refinement layer together to obtain the final depth picture. The size of the input picture can be 224×224×3, and the size of the depth picture can be 112×152×1. The depth picture is interpolated in two dimensions to obtain depth information consistent with the size of the input picture.

[0132] In an exemplary embodiment of the present application, the input of the monocular ranging module can be the result of segmentation of all Class 1 pictures, and the three parts of all Class 1 pictures marked as selected are put into the first neural network for calculation, and then the depth maps of the three pictures are merged according to the ranging result of the middle half of the picture; specifically, the depth of the overlapping parts can be averaged, and the non-overlapping parts can be offset by calculating the offset between the calculated value of the overlapping parts and the final average value. The first neural network is used to measure and judge the depth of the point in the picture from the camera device.

[0133] In an exemplary embodiment of the present application, the object detection module can utilize an artificial intelligence algorithm, namely the latest yolo-v3 algorithm, utilize Darknet-v3 for feature extraction, and utilize methods such as anchor box for corresponding target detection.

[0134] In an exemplary embodiment of the present application, the object detection module uses Darknet-v3 as the backbone neural network, uses the COCO dataset as training data, and removes some object information that has a low probability of appearing in the application scenario of the embodiment of the present application. For example, the YOLO-v3 target detection algorithm can be used as the upper-layer object detection method. Since the final required granularity is not high, the number of anchor boxes can be specialized to 3, and the image can be divided into a 5×5 grid. First, the backbone neural network is placed in the ImageNet dataset for object category judgment training, then the first few layers of the network are frozen, and the COCO dataset is placed for target detection training, and finally the first few layers of the network are unfrozen for overall training.

[0135] In an exemplary embodiment of the present application, the input of the object detection module is the segmentation result of all the Class 1 images. The three parts of all the Class 1 images marked as selected are put into the second neural network for operation, and then all the identified objects are converted to coordinates, converted back to the original image coordinates, and then the object frame pairs with relatively large intersections are found, and the one with a smaller probability is deleted. The second neural network realizes the detection and position measurement of common objects in the object set appearing in the image.

[0136] In an exemplary embodiment of the present application, a type 1 image is cut into three overlapping areas: upper, middle, and lower, and monocular ranging and object detection calculations are performed separately, thereby improving recognition accuracy.

[0137] In an exemplary embodiment of the present application, the place detection module can use the latest ResNet-152 neural network and the Place365 data set to predict the most likely multiple place scenarios, such as five scenarios.

[0138] In the exemplary embodiment of the present application, the place detection module uses ResNet-152 as the backbone neural network, uses the Place365 data set for training, and deletes the images and labels in the data set that are unlikely to appear in the application scenario of the embodiment of the present application. The network is first put into the ImageNet data set for object category judgment training, then the first few layers of the network are frozen, put into the Place365 data set for training, and finally the first few layers are unfrozen for overall training.

[0139] In an exemplary embodiment of the present application, the input of the location detection module is the original image before all the 1-type images are segmented, and all the original images marked as the selected 1-type images are input into the third neural network for calculation, and the most likely n (such as five) locations and their probabilities are calculated. The third neural network is used to determine the location of the shooting device of the image.

[0140] In an exemplary embodiment of the present application, the cloud server can send the calculation results of the three modules, namely, the monocular ranging module, the object detection module and the location detection module, back to the edge server respectively, or integrate and package the calculation results of the four modules and send them to the surrounding calculation module. For example, the following information can be included:

[0141] The depth map of the monocular ranging module has a size of 224×448;

[0142] The object box of the object detection module, i.e. the coordinates of the upper left corner, width, height, and type of the object;

[0143] The five most likely place names and their probabilities from the place detection module.

[0144] In an exemplary embodiment of the present application, the scene description generation module can take a video as input and output a description of what happened in the video. It uses ResNet-152 as an encoder to extract features from the input video frame by frame, uses SA-LSTM as a decoder to process each input frame with an attention mechanism, and inputs it into a multi-layer long short-term memory neural network unit (LSTM Unit) arranged in a sequence to generate a picture description.

[0145] In an exemplary embodiment of the present application, the scene description generation module uses ResNet-152 as the encoder neural network and a long short-term memory network with an attention mechanism as the decoder neural network. For the training of the encoder, the encoder is placed in the ImageNet data set for training of object category judgment, and the encoder is frozen in the subsequent training process without back propagation correction. The last layer of the encoder neural network is removed so that the encoder outputs the feature vector of the picture instead of the category information. The input video is frame-processed, and each frame is input into the encoder neural network to obtain the corresponding feature vector matrix, and the matrix is ​​processed by the attention mechanism. For the decoder, part of its input is the feature vector matrix after the attention mechanism is processed, and the other part is the sentence that has been generated. The task of the decoder is to predict the next word of the sentence based on these two kinds of information, which increases the robustness of the decoder. After a certain word has a prediction deviation, the following words can also reduce the deviation as much as possible. When applied, the next word is selected by a beam search method.

[0146] In an exemplary embodiment of the present application, the scene description generation module uses a pre-trained fourth neural network to describe what happens in a video in natural language. Specifically, all unsegmented images of category 1 and all category 2 images can be packaged into a video at a certain time interval and input into the fourth neural network for calculation.

[0147] In an exemplary embodiment of the present application, the scene description generation module can be set in the edge server, and the edge server can input all the class 1 pictures and class 2 pictures into the encoder of the scene description generation module every 30 seconds to obtain the corresponding 30×30 feature vectors, and then input the feature vectors into the decoder of the scene description generation module to obtain the initially generated scene description. This calculation can occur in parallel with the calculation of the monocular ranging module, the object detection module, and the place detection module.

[0148] In an exemplary embodiment of the present application, the edge server may integrate the calculated scene description and the calculation results received from the cloud server, and generate a specific auxiliary interaction statement.

[0149] In an exemplary embodiment of the present application, in the surrounding computing module, the auxiliary information implemented to the user may be first determined according to the data returned by the core computing subsystem.

[0150] In an exemplary embodiment of the present application, the auxiliary information may include any one or more of the following: distance warning, object prompt, location change prompt, scene description, and active guidance;

[0151] Wherein, the distance alarm is used to issue a reminder to the assisted person when the distance between the assisted person and the detected object is less than or equal to a preset distance threshold;

[0152] The object prompt is used to describe the object information corresponding to the detected object to the assisted person; the object information may include: name and / or spatial coordinates;

[0153] The location change prompt is used to prompt the assisted person that the current location has changed and provide the assisted person with multiple possible locations;

[0154] The scene description is used to describe the dynamic scene in the captured video in words;

[0155] The active guidance is used to determine obstacles based on the object depth information obtained in the distance detection, and to provide road direction guidance to the assisted person in the form of voice.

[0156] In an exemplary embodiment of the present application, before implementing the above auxiliary functions, the coordinate system used in the embodiment of the present application can be first determined to facilitate the determination of the spatial coordinates of the object and the description of the above auxiliary information.

[0157] In an exemplary embodiment of the present application, the coordinate system where the spatial coordinates of the object are located may take the center of gravity of the assisted person as the coordinate origin, and the coordinate system may include: a depth coordinate, a horizontal coordinate, and a vertical coordinate;

[0158] The direction in which the assisted person is facing in the horizontal direction may be the positive direction of the depth coordinate, the direction in which the left hand side of the assisted person is located in the horizontal direction may be the positive direction of the horizontal coordinate, and the vertically upward direction may be the positive direction of the vertical coordinate;

[0159] The horizontal axis is divided into segments a, the vertical axis is divided into segments b, and the vertical axis is divided into segments c, each segment having a corresponding natural language name;

[0160] On the horizontal plane of the coordinate system, the area 180 degrees in front of the person being assisted is divided into d sectors, each sector has a corresponding natural language name; a, b, c, d are positive integers.

[0161] In the exemplary embodiment of the present application, a special coordinate description method is adopted, which can divide the horizontal direction into five segments, the vertical direction into three segments, and the front-to-back direction into three segments, and each segment is named in natural language. The coordinate change can also be segmented, and the horizontal, vertical, and front-to-back directions are divided into two segments (one segment each in the positive and negative directions), and each segment is also named in natural language. The ray angle (i.e., azimuth) is divided into five segments, with the vertical as the center and two segments on each side, named in natural language.

[0162] In an exemplary embodiment of the present application, for example, the horizontal coordinate of the input image can be divided into five segments, the vertical coordinate into three segments, and the depth coordinate into three segments, each of which has a corresponding natural language name. The horizontal, vertical, and depth coordinate changes are divided into two segments, namely, one segment for positive changes and one segment for negative changes. The 180-degree direction in front of the user is divided into five segments, one segment each for the left, right, oblique left, oblique right, and front.

[0163] In an exemplary embodiment of the present application, Figure 4 As shown, if the horizontal direction of the 224×448 image is divided into 5 equal parts, they are called left, left, middle, right, and right; the vertical direction is divided into 3 equal parts, called ground, middle, and air; the front and back directions are divided into 3 parts with 2 meters and 5 meters as the dividing points, called in front, near, and far. The changes in the horizontal coordinate (i.e. the horizontal coordinate) are called moving to the left and moving to the right; the changes in the vertical coordinate (i.e. the vertical coordinate) are called moving up and moving down; the changes in the depth coordinate are called approaching, moving away, and not moving. The direction in front is divided into five parts, called left, left front, forward, right front, and right.

[0164] In an exemplary embodiment of the present application, before implementing the above auxiliary functions, objects can also be divided into two categories, one is that which can be suspended in the air, and the other is that which cannot be suspended in the air. Objects that can be suspended in the air are described by describing the three-dimensional coordinates of horizontal (horizontal coordinate), vertical (vertical coordinate), and front and back (depth coordinate), while objects that cannot be suspended in the air are described by only describing the two-dimensional coordinates of horizontal and front and back.

[0165] In an exemplary embodiment of the present application, the surrounding calculation module can use the vertical coordinate of the object information detected during object detection to link it with the depth information of its corresponding position, and use the perspective algorithm to correct the vertical coordinate. Among them, for objects classified as non-suspended objects, their vertical coordinates are corrected to empty, that is, their vertical coordinates are not described during description. Through this embodiment, the influence of the perspective effect on the three-dimensional coordinate detection of the object can be reduced.

[0166] In an exemplary embodiment of the present application, before implementing the above auxiliary functions, the image can also be gridded. For example, according to the above-mentioned coordinate segmentation, the input image can be divided into 15 grids of 5×3. When it comes to interaction or information output involving the entire picture, each grid is processed as a whole to facilitate content description during auxiliary prompts.

[0167] In an exemplary embodiment of the present application, the determining of the auxiliary information for the assisted person according to the calculation result of the core calculation may include one to five of the following contents:

[0168] 1. Distance alarm

[0169] The depth information detected by the distance detection is determined as a first distance between any first object on the picture and the assisted person, the first distance is compared with the distance threshold, and when the first distance is less than or equal to the distance threshold, the distance alarm is generated.

[0170] In an exemplary embodiment of the present application, the distance alarm function can simultaneously alarm through dual channels of voice prompt and device vibration.

[0171] In an exemplary embodiment of the present application, the distance warning function can be based on the above-mentioned gridding method, and several (such as two) grids with the smallest distance to the assisted person are selected, and their distances are recorded as the two nearest detection distances. If the nearest detection distance is less than the set distance threshold, a distance reminder can be made, and the reminder method can describe the coordinates of the corresponding grid and the nearest detection distance according to the above-mentioned coordinate description method. When the nearest detection distance is too close, it can be supplemented by a reminder of vibration of the handheld device. In order to further enhance the user's sense of security, if there is no distance alarm information at regular intervals, a descriptive reminder of the nearest detection distance and the corresponding grid coordinates can be made.

[0172] In an exemplary embodiment of the present application, the distance alarm function can be triggered every 5 seconds according to the calculation interval set by the core computing subsystem. Calculate the minimum distance of each grid of the 5×3 grid. If the minimum distance is extreme and the surrounding gradient is large, remove it and recalculate. Taking the distance threshold of 0.3 meters as an example, take the two grids with the smallest minimum distance. If one is less than 0.3, describe it according to the coordinate description method, and assist with mobile phone vibration. If the distance between the two grids is less than 0.3, the coordinates of the two grids are described according to the coordinate description method, and the mobile phone is also vibrated. If both distances are greater than 0.3, no reminder will be made. If no reminder occurs within 30 seconds, the grid with the smallest distance will be described according to the coordinate description method, and the mobile phone will not be vibrated. The sentence form of the prompt is such as "There is an obstacle approaching at a certain coordinate (the natural language coordinate) in front of you."

[0173] 2. Object Prompts

[0174] Determine a first name of the first object according to the type of object detected in the object detection; determine a first height of the first object according to the object boundary information detected in the object detection; determine whether the first object can be suspended in the air according to the first name; determine a first spatial coordinate of the first object according to the first distance, the first height and the first orientation of the first object relative to the assisted person detected in the object detection; correct the first spatial coordinate of the first object according to the information of whether the first object can be suspended in the air; generate a prompt content of the object prompt according to the first name, the corrected first spatial coordinates and the suspended state of the first object.

[0175] In an exemplary embodiment of the present application, generating the prompt content of the object prompt according to the first spatial coordinates may include:

[0176] When the first spatial coordinate is located on the horizontal coordinate, the vertical coordinate or the ordinate, forming a description content about the first spatial coordinate in the prompt content according to the natural language name corresponding to the segment where the first spatial coordinate is located;

[0177] When the first spatial coordinate changes, the description content of the first spatial coordinate is updated in real time according to the change of the first spatial coordinate.

[0178] In an exemplary embodiment of the present application, when an object appears for the first time, the horizontal, vertical and deep coordinates of the object and the object category of the object are described according to the coordinate segmentation method, wherein objects with the same coordinates, that is, objects with the same coordinates after segmentation, can be combined during description and can be described simultaneously in the same sentence. When the same object appears for the second time, the change in the coordinates and the object type can be described in segments according to the change in the coordinates.

[0179] In an exemplary embodiment of the present application, the object description (ie, object prompt) function can also perform overall picture description, and perform distance and object description for each 5×3 grid.

[0180] In an exemplary embodiment of the present application, the object prompt function can be triggered every 5 seconds according to the calculation interval set by the core computing subsystem. The prompt sentence is in the form of "there is a certain object at a certain coordinate (the natural language coordinate) in front of you".

[0181] 3. Reminder of location change

[0182] Retrieving n possible location scenes predicted last time according to the location detection, and comparing first location information obtained during the current location detection process with second location information of the n possible location scenes predicted last time; when a difference rate between the first location information and the second location information is greater than or equal to a preset difference rate threshold, determining that the current location has changed, and re-predicting n possible location scenes according to the first location information; and generating prompt content of the location change prompt according to the information of the change in the current location and the re-predicted n possible location scenes.

[0183] In an exemplary embodiment of the present application, when there are differences between the most likely n (e.g., five) locations given by the location detection result and the current location, and similar differences are produced in the identification information of multiple pictures, it can be determined that the location has changed, and a corresponding prompt is given. The prompt content is not the most likely location, but provides multiple possible locations for the user to judge based on the background information.

[0184] In the exemplary embodiment of the present application, the location change prompt function adopts a multi-frame judgment method, and only when the location recognized in many frames of pictures changes, the location change prompt is performed, and multiple most likely locations are given each time the location changes. During the training process, the places in the Place365 data set that are unlikely to appear in the application scenario of the embodiment of the present application are discarded.

[0185] In an exemplary embodiment of the present application, each time a location detection is performed, the five most likely locations and their probabilities can be recorded. According to the calculation interval of the core computing subsystem, a check can be performed every 5 seconds. If the most likely five locations detected by the location detection module for five consecutive times calculated by the core computing subsystem are very different from the currently recorded location (i.e., the n possible locations predicted and recorded in the previous interval), the current location is updated, and the two locations with the highest probability among the five times are selected and prompted to the user, and the prompt format is such as "You are now at such and such a place".

[0186] 4. Scenario Description

[0187] The voice description information obtained in the scene description generation calculation is retrieved, and the voice description information is used as the description content of the scene description.

[0188] In an exemplary embodiment of the present application, the scene description function can simply modify the output sentence of the scene description generation module to make it more like a conversation. The scene description sentence generated by the core computing subsystem can be slightly modified and output, and the purpose of the modification is mainly to make the sentence more friendly and conversational.

[0189] In an exemplary embodiment of the present application, the scene description function may be triggered every 30 seconds according to the calculation interval of the core computing subsystem. The sentence generated by the scene description generation module is slightly modified to a format such as "something happened in front of you".

[0190] In the exemplary embodiment of the present application, artificial intelligence technology is used to synthesize emotional voices, eliminating the cold and mechanical feeling. In addition, the sentences generated by the surrounding calculation module tend to be spoken, using the dialogue style to create a sense of companionship.

[0191] 5. Active Guidance

[0192] According to the depth information detected in the distance detection, it is determined whether there is a continuous segment around the assisted person whose depth is greater than or equal to a preset depth threshold. When the continuous segment exists, the continuous segment is determined as a feasible route as the guidance direction of the active guidance.

[0193] When the guiding direction is located within the sector, the content of the active guiding direction in the prompt content is formed according to the natural language name corresponding to the sector where the guiding direction is located.

[0194] In an exemplary embodiment of the present application, the active guidance function can actively provide the user with direction guidance of a wide road based on the current depth information and obstacle judgment information.

[0195] In an exemplary embodiment of the present application, after the active guidance function is turned on, a judgment can be made based on the depth map. The evaluation function calculation method of a vertical line (original coordinates, not the natural language description coordinates) is the gradient below the ground line (the set vertical coordinate threshold) (taking into account the perspective effect of the ground), the closest distance above the ground line, and the additional cost of turning. Select a horizontal coordinate interval whose length is greater than a set threshold (i.e., the minimum width of the road) so that the maximum evaluation function of all vertical lines therein is minimized. Determine the direction of this interval relative to the user, and use the natural language description of the direction to feedback to the user. After selecting a direction, in the next calculation, the additional cost of turning the vertical line opposite to the direction will be reduced (can be less than 0), and the additional cost of turning the vertical line in the same direction will increase, and try to guide the user to turn back after turning, so that the user's forward direction can be not changed as a whole.

[0196] In an exemplary embodiment of the present application, the active guidance function can be activated when the user walks to a place with many obstacles. According to the depth information provided by the core computing subsystem, determine which direction in front of the user has a continuous segment with a deeper depth, and identify it as a possible route. Guide the user to move in that direction, and the direction information can be conveyed to the user according to the aforementioned direction segmentation method. When a possible route cannot be detected in the picture, guide the user to turn left or right and guide in a new direction. After the user turns left or right, the guidance weight in the corresponding opposite direction becomes higher, and try to ensure that the user walks in a straight line.

[0197] In an exemplary embodiment of the present application, the method may further include:

[0198] Collecting the input assistance help instruction, and determining the assistance information for the person being assisted according to the assistance help instruction;

[0199] The auxiliary help instruction is a signal used to indicate that the person being assisted needs help; the auxiliary information corresponding to the auxiliary help instruction includes: sending help information.

[0200] In an exemplary embodiment of the present application, when the assisted person needs help, he or she can actively seek help from people around him or her through the above-mentioned scheme.

[0201] The person being assisted can enable the auxiliary help function through the mobile terminal and input the auxiliary help command through the input collection module. When the mobile terminal receives the auxiliary help command, it can attract the attention of people around by sound and / or light signals through the hardware in the help module, so as to achieve the purpose of seeking help. After obtaining help, the person being assisted can turn off the auxiliary help function through the input collection module again.

[0202] In an exemplary embodiment of the present application, the method may further include: determining the auxiliary information to be used according to the current visual assistance mode;

[0203] The visual assistance mode may include any one or more of the following: walking mode, exploration mode, close-up mode and standby mode.

[0204] In an exemplary embodiment of the present application, determining the auxiliary information to be used according to the current visual assistance mode may include:

[0205] When the user is currently in the walking mode, the user is guided in the road direction according to the active guidance, and objects that need to be reminded detected by the object detection are screened out, and the objects that need to be reminded are described according to the object prompt;

[0206] When the exploration mode is currently in progress, a spatial overall description is performed for the currently collected picture at a preset time interval; wherein the picture is divided into a plurality of grids, and the description content includes any one or more of the following: a scene description of each grid, a spatial coordinate description of each grid, a description of the average distance between each grid and the assisted person, and a description of the object information of the objects contained in each grid;

[0207] When the close-up mode is currently in place, the object prompt is executed, wherein when the number of objects of the same type detected is greater than or equal to a preset number threshold, the detected objects of the same type are described together in the prompt content of the object prompt, and when the number of objects of the same type detected is less than the number threshold, the detected objects of the same type are described separately in the prompt content of the object prompt; and the prompt content only includes a description of the horizontal coordinate and / or the vertical coordinate, and does not include a description of the depth coordinate;

[0208] When the current mode is in the standby mode, only the current visual assistance mode is uploaded as the standby mode, so that the calculation is stopped according to the standby mode.

[0209] In an exemplary embodiment of the present application, the user's mode configuration can be read from the mobile terminal, specifically, it can include the set visual assistance mode (walking mode, exploration mode, close-up mode and standby mode) and whether active navigation is turned on. The user's mode configuration and the calculation result of the core computing subsystem can be sent to the surrounding computing module together.

[0210] In an exemplary embodiment of the present application, the walking mode can be applied when the user has a clear destination and needs to reach the destination quickly and safely. In this mode, the distance alarm, object prompt, and location change prompt functions in the surrounding calculation module can all be activated.

[0211] In an exemplary embodiment of the present application, the exploration mode can be applied when the user needs to know more about the current state of the space. The main purpose of this mode is to enhance the user's spatial cognition and allow the user to have more detailed concepts about the current space in his mind. In this mode, the distance alarm, object prompt, location change prompt and dynamic scene description functions in the surrounding calculation module can all be activated.

[0212] In an exemplary embodiment of the present application, the near-view mode can be applied when the user remains basically still and there are relatively still obstacles in front of him that will not cause danger, such as on a crowded bus or in front of a desk. In this mode, only the object prompt function in the surrounding calculation module may be activated.

[0213] In an exemplary embodiment of the present application, the standby mode can be applied when the user arrives at a familiar place or does not need visual assistance temporarily. In this mode, the camera on the mobile terminal will not take pictures, and the only data transmission can be uploading the standby mode to the core computing subsystem and the surrounding computing modules to stop computing.

[0214] In an exemplary embodiment of the present application, it is determined whether the current mode is walking mode, close-up mode, or exploration mode. If the current mode is walking mode, objects that are likely to cause danger and emit noticeable sounds are screened out from the objects identified by the core computing subsystem, and described in accordance with the aforementioned object description method. If the current mode is close-up mode, objects that are identified by the core computing subsystem and are identified in smaller numbers are described in accordance with the aforementioned object description method, but only the horizontal and vertical coordinates and their changes are described, and the depth coordinates and their changes are ignored; if a certain object is identified in larger numbers, they are combined and described, that is, the coordinates of all blocks that identify the object and the object type are described. If the current mode is exploration mode, a spatial overall description is performed at regular intervals, that is, all fifteen grids are described, and the content can be its position, average distance, and the objects that appear in the grid, which can be described in accordance with the aforementioned object description method until the next spatial overall description.

[0215] In an exemplary embodiment of the present application, for the aforementioned object prompt function, different processing may be performed for different visual assistance modes (walking mode, exploration mode, close-up mode, and standby mode).

[0216] In an exemplary embodiment of the present application, for walking mode, the object prompt function can screen out objects that may cause danger and emit noticeable sounds among the identified objects, and find their corresponding distances in the depth map. A dynamic target detection algorithm can be applied. If it is determined that the object appears for the first time, the object coordinates and distance are converted to the aforementioned coordinate description method and output together with the object category. New objects in the same grid can be merged and described in the same sentence without repeating the coordinate description; if it is determined that the object has appeared and been described before, the object coordinate change and distance change can be converted to the coordinate change description method, and objects with the same coordinate change can be merged and described in the same sentence without repeating the coordinate change.

[0217] In an exemplary embodiment of the present application, for the exploration mode, an overall spatial description may be performed every 30 seconds or when there is an obvious nonlinear change in the depth image. The steps of overall spatial description may include: from bottom to top and from left to right, describing the coordinates of fifteen grids, the average depth, and the categories of objects appearing therein according to the aforementioned coordinate description method. When the overall spatial description is not performed, the objects that appear for the first time or have appeared are described separately according to the above method. In this mode, object screening is not performed, and all identified objects can be described.

[0218] In the exemplary embodiment of the present application, for the close-up mode, the objects that appear for the first time or the objects that have appeared can be described separately according to the above method, and no object screening is performed. If a certain object is recognized many times in the same picture, it is combined and described in a way that all recognized coordinates are described according to the coordinate description method, and the object category and number are described.

[0219] In an exemplary embodiment of the present application, the surrounding calculation module may send all generated description sentences to the speech synthesis module, and both the surrounding calculation module and the speech synthesis module may be arranged in the edge server.

[0220] In an exemplary embodiment of the present application, the speech synthesis module can call an artificial intelligence algorithm to generate a slow and gentle human voice, send it to the mobile terminal, and the voice interaction module in the mobile terminal plays the received sound file.

[0221] In an exemplary embodiment of the present application, the speech synthesis module can generate speech using a long short-term memory network, which is divided into an encoder part and a decoder part. The encoder part inputs a sentence processed by an attention mechanism, and the decoder part inputs the next predicted syllable. For example, a mature speech synthesis interface provided by open source software can be used for synthesis.

[0222] In the exemplary embodiments of the present application, several embodiments are given for the setting method for the visual assistance mode.

[0223] In an exemplary embodiment of the present application, the four modes of walking mode, exploration mode, close-up mode and standby mode can be arranged in a ring, such as "walking mode - exploration mode - close-up mode - standby mode - walking mode", and the mode switching setting can be performed using a simple input method of switching to an adjacent mode.

[0224] In an exemplary embodiment of the present application, the volume up and down keys of an Android phone can be used as mode switching buttons, where the volume up key indicates switching to the previous mode, and the volume down key indicates switching to the next mode.

[0225] In an exemplary embodiment of the present application, the volume up and down keys of an Android phone can be used as buttons for turning on the active guidance function. For example, pressing the two buttons at the same time means turning on the active guidance function.

[0226] In an exemplary embodiment of the present application, the software system on the Android phone sends the user's mode configuration (including whether the current visual assistance mode and active navigation are turned on) and the photos taken to the edge server at a rate of 30 times per second. After the edge server detects that the user's mode configuration has changed, it synthesizes a prompt voice and returns it to the phone, such as "Current mode: so-and-so" or "Active guidance: on (or off)".

[0227] In an exemplary embodiment of the present application, a method of intelligently switching the visual assistance mode and switching active navigation may also be adopted:

[0228] 1. For the most common switching between walking mode and exploration mode, users can still switch manually because these two modes are related to user needs.

[0229] 2. When the acceleration sensor and gyroscope data uploaded by the mobile terminal remain stable for a period of time, or the depth map data remains low for a long time, the close-up mode can be turned on. When both conditions are not met, the close-up mode is automatically exited and switched to the walking mode by default.

[0230] 3. When the object detection system detects a large number of objects and the corresponding distances are close, the active guidance function is turned on. When this condition is not met, the active guidance function is automatically exited.

[0231] 4. When the GPS (Global Positioning System) data shows that the user has reached a location where the user often switches to standby mode, it is considered that the user is in a familiar place and does not need the assistance of the embodiment of the present application. If this condition is not met, or the user manually turns it on, the standby mode is exited and the default switch is to walking mode.

[0232] 5. Use the up and down selection method to configure the mode. For mobile terminals with buttons, users can press the button to indicate switching to the next mode. For mobile terminals without buttons, users can use specific gestures (i.e., specific shaking and displacement of the mobile terminal) to indicate switching to the next mode.

[0233] 6. The active guidance can be turned on or off by intelligent recognition or by the user. For mobile terminals with buttons, the user can turn on or off the active navigation by pressing another button. For mobile terminals without buttons, the user can use another specific gesture to switch to the next mode.

[0234] In an exemplary embodiment of the present application, for a mode configuration change actively initiated by a user, the user may be notified of the change result in voice form after the change.

[0235] In the exemplary embodiment of the present application, in summary, the software system of the embodiment of the present application may include a core computing subsystem, a surrounding computing module and a speech synthesis module. The core computing subsystem uses artificial intelligence technology to perform preliminary processing on the input image. The surrounding computing module further processes these preliminary processing results and generates corresponding description sentences from the user's psychological feelings. The speech synthesis module converts the generated sentences into slow and gentle human voices through mature artificial intelligence algorithms. The hardware system of the embodiment of the present application may include a mobile terminal, an edge server and a cloud server. The mobile terminal can directly use an ordinary smart phone with a camera, a gyroscope, and an acceleration sensor. The edge server can use edge computing technology in the 5G, 6G and other eras to handle operations with large amounts of transmitted data and complicated steps. The cloud server accepts the data processed and screened by the edge server and performs more core operations.

[0236] In the exemplary embodiments of the present application, the embodiments of the present application at least include the following advantages:

[0237] 1. Compared with manual guide devices such as canes, it is more active. Users do not need to actively use their hands to detect the area in front of them, which reduces the granularity of detection and the physical load of users. The lack of active detection also makes it easier for the device to handle objects that are not easy to detect through contact detection or that will cause more trouble after contact.

[0238] 2. Compared with blind sticks or mechanical dogs that are enhanced by sonar devices such as EyeCane, it has a larger sensing range. The main sensing range of these devices is near the ground, while the embodiment of the present application can detect not only objects on the ground, but also objects suspended in the air.

[0239] 3. Compared with other blind guide devices that rely on sonar devices for input, it has the characteristic of being able to perceive the type of objects. Other devices that rely on sonar input can only measure the distance of the reflective surface, but cannot determine the type or name of these objects. The embodiment of the present application utilizes an artificial intelligence algorithm, which can determine the type of the object while measuring the distance of the object, thereby increasing the user's cognition of the space in which they are located.

[0240] 4. Compared with other devices that describe spatial scenes for the blind, while covering their functions, more basic functions are added, such as distance alarm, etc. Distance alarm is the most basic function, which can prevent the visually impaired from bumping into obstacles and getting injured while walking, and is the most basic function to enhance the sense of security of the visually impaired.

[0241] 5. Pay more attention to the design of the interactive process. Some guide devices based on computer vision algorithms only give digital information. This way of presenting information does not conform to the rules of interactive design. Numbers cannot quickly form intuitive location information in people's minds, and a series of complex processing is required, which increases the user's mental workload and reduces the real-time nature of information processing. Over time, due to the continuous accumulation of the load of these processing processes, this device becomes difficult for users to use. The embodiment of the present application adopts an interactive design method that is more in line with the rules, and adopts a coordinate position description method using natural language to speed up the cognitive process.

[0242] 6. Focus on improving the sense of security for the visually impaired. The visually impaired group often panics due to various unwarranted noises during travel. For example, knowing that there are some objects nearby but not knowing their distance, they often worry about whether they will collide with the objects and cause danger. Reduced sense of security will lead to a decrease in the willingness of the visually impaired to travel and a decrease in their quality of life. The embodiment of the present application attempts to solve this problem, and to enhance the experience of the visually impaired during travel and improve their quality of life.

[0243] 7. Strive to improve the visually impaired group's cognition of the space they are in, and describe the current scene in detail through object coordinate description and grid depth description. Provide a holistic scene description through location change prompts and dynamic scene descriptions. The combination of the two forms a picture with richer content, which is presented to the visually impaired group and increases the amount of information they obtain.

[0244] 8. Give users a sense of companionship. When generating sentences, try to adopt the form of simulated dialogue. The speech synthesis module of the embodiment of the present application abandons the common mechanical synthesis of human voice and adopts artificial intelligence algorithm for human voice synthesis. The synthesized voice is more emotional, slow and gentle, which increases the sense of companionship and makes users feel less lonely.

[0245] 9. According to the survey, when blind people walk into complex obstacles, they often bump into walls everywhere and cannot find a wide road, which leads to panic. For example, if you accidentally walk into a group of shared bicycles parked in a messy manner on the side of the road, you may have to go through a long time of groping and multiple collisions before you can get out of the group of bicycles. The embodiment of the present application provides an active guidance function. On the basis of increasing the perception range, it uses the perceived data to search for a wider road, guide the user to move in that direction, and get out of the scene of complex obstacles as soon as possible while reducing the number of explorations.

[0246] 10. Use artificial intelligence technology to process underlying visual information, convert the processed information into a form that is easy for the visually impaired to accept, and use edge computing in the 5G era to improve processing efficiency and enhance users' sense of security and spatial cognition when traveling.

[0247] The present application also provides a visual assistance system 1, such as Figure 5 As shown, it may include: an image data acquisition module 101, a core calculation module 102, an auxiliary information determination module 103 and a speech processing module 104;

[0248] The image data acquisition module 101 is configured to acquire image data of the environment;

[0249] The core computing subsystem 102 is configured to perform core computing based on the image data; the core computing includes any one or more of the following: distance detection, object detection, location detection, and scene description generation;

[0250] The auxiliary information determination module 103 is configured to determine auxiliary information for the assisted person according to the calculation result of the core calculation;

[0251] The speech processing module 104 is configured to convert the auxiliary information into interactive sentences and transmit the interactive sentences to the assisted person in the form of speech.

[0252] In an exemplary embodiment of the present application, the core computing subsystem may include: a distance detection module, an object detection module, a location detection module, and a scene description generation module; the speech processing module may include a speech synthesis module and a speech interaction module;

[0253] Among them, the distance detection module, the object detection module and the location detection module can be set on a cloud server, the scene description generation module, the auxiliary information determination module and the speech synthesis module can be set on an edge server, and the voice interaction module can be set on a mobile terminal.

[0254] In an exemplary embodiment of the present application, the visual assistance system may further include: an input acquisition module and a help-seeking module;

[0255] The input collection module may be configured to collect input assistance help instructions; the assistance help instructions are signals used to indicate that the person being assisted needs help;

[0256] The assistance module may be configured to determine assistance information for the person being assisted according to the assistance assistance instruction; the assistance information corresponding to the assistance assistance instruction includes: sending assistance information.

[0257] In an exemplary embodiment of the present application, the input acquisition module can be configured to obtain the input information of the assisted person to the system (the input information may include the above-mentioned auxiliary help instructions) for system adjustment, and the input information may include one or more of the following: switching modes, turning on or off active guidance, and turning on or off auxiliary help.

[0258] In an exemplary embodiment of the present application, the help-seeking module may be configured to convert the help-seeking information into one or more of a light signal and a sound signal, which are displayed on the device and communicated to people around.

[0259] In an exemplary embodiment of the present application, for example, when a user needs to seek help from people around him / her, he / she can notify the system by pressing a button on the mobile terminal, and the system will ask for help from people around him / her by activating the flashlight of the mobile terminal, playing voice help information, etc.

[0260] In an exemplary embodiment of the present application, the help information may be a 1-bit data, that is, it is only used to indicate whether the person being assisted needs to ask for help from others. The specific content of the help can be expressed by the person being assisted to others, and the mobile terminal is only responsible for attracting the attention of people around. The present application also provides a visual assistance device 2, such as Figure 6 As shown, it may include a processor 21 and a computer-readable storage medium 22, wherein the computer-readable storage medium 22 stores instructions, and when the instructions are executed by the processor 21, any one of the above-mentioned visual assistance methods is implemented.

[0261] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A visual assistance method, characterized in that: include: Collect image data of the environment; The image data includes: pictures and / or videos; Performing core calculations based on the image data; the core calculations include any one or more of the following: distance detection, object detection, location detection, and scene description generation; Determining auxiliary information for the assisted person according to the calculation result of the core calculation; Converting the auxiliary information into interactive sentences, and transmitting the interactive sentences to the assisted person in the form of voice; The auxiliary information includes any one or more of the following: distance alarm, object prompt, location change prompt, scene description and active guidance; Wherein, the distance alarm is used to issue a reminder to the assisted person when the distance between the assisted person and the detected object is less than or equal to a preset distance threshold; The object prompt is used to describe the object information corresponding to the detected object to the assisted person; the object information includes: name and / or spatial coordinates; The location change prompt is used to prompt the assisted person that the current location has changed and provide the assisted person with multiple possible locations; The scene description is used to describe the dynamic scene in the captured video in words; The active guidance is used to determine obstacles based on the object depth information obtained in the distance detection, and provide road direction guidance to the assisted person in the form of voice; The determining of the auxiliary information for the assisted person according to the calculation result of the core calculation includes: Determining the depth information detected by the distance detection as a first distance between an arbitrary first object on the picture and the assisted person, comparing the first distance with the distance threshold, and generating the distance alarm when the first distance is less than or equal to the distance threshold; Determine a first name of the first object according to the type of object detected in the object detection; determine a first height of the first object according to the object boundary information detected in the object detection; determine whether the first object can be suspended in the air according to the first name; determine a first spatial coordinate of the first object according to the first distance, the first height, and a first position of the first object relative to the assisted person detected in the object detection; correct the first spatial coordinate of the first object according to the information of whether the first object can be suspended in the air; generate a prompt content of the object prompt according to the first name, the corrected first spatial coordinate, and the suspended state of the first object; Retrieving n possible locations predicted last time according to the location detection, and comparing first location information obtained during the current location detection process with second location information of the n possible locations predicted last time, and when a difference rate between the first location information and the second location information is greater than or equal to a preset difference rate threshold, determining that the current location has changed, and re-predicting n possible locations according to the first location information; generating prompt content of the location change prompt according to the information of the change in the current location and the re-predicted n possible locations; Retrieving the voice description information obtained in the scene description generation calculation, and using the voice description information as the description content of the scene description; determining, based on the depth information detected in the distance detection, whether there is a continuous segment around the assisted person whose depth is greater than or equal to a preset depth threshold, and if the continuous segment exists, determining the continuous segment as a feasible route as the guidance direction of the active guidance; The step of correcting the first spatial coordinate of the first object according to the information of whether the first object can be suspended in the air comprises: The vertical coordinate in the object information of the first object and the depth information of the corresponding position of the first object are used to correct the vertical coordinate using a perspective algorithm; wherein, when the first object is classified as an object that cannot be suspended in the air, the vertical coordinate of the first object is corrected to empty.

2. The visual assistance method according to claim 1, characterized in that: The performing core calculation according to the image data comprises: Inputting the collected picture into a pre-trained first neural network, determining a feature object in the picture through the first neural network, and measuring the depth between the feature object and a shooting device of the picture to achieve the distance detection; Inputting the collected picture into a pre-trained second neural network, detecting whether there is an object and the type of the object in the picture through the second neural network, and measuring the position of the object to achieve the object detection; the position of the object includes the orientation of the object relative to the person being assisted; Inputting the collected picture into a pre-trained third neural network, detecting the scene in the picture through the third neural network, and predicting n possible scenes, where n is a positive integer, to achieve the location detection; The captured video is input into a pre-trained fourth neural network, and voice description information of events in the video is generated by the fourth neural network to achieve the scene description generation.

3. The visual assistance method according to claim 2, characterized in that: The first neural network includes a SeNet-154 neural network, and uses the NYU-v2 dataset as a training dataset; The second neural network includes a Darknet-v3 neural network, which implements object detection based on the yolo-v3 algorithm; The third neural network includes a ResNet-152 neural network, and uses the Place365 data set as a training data set; The fourth neural network includes: an encoder ResNet-152 and a decoder SA-LSTM.

4. The visual assistance method according to any one of claims 1 to 3, characterized in that: The method further includes: the image data includes a picture; before performing core calculations based on the image data, preprocessing the picture; The preprocessing of the picture comprises: Screen out qualified images from all collected images and remove unqualified images; and, The qualified image is divided into a plurality of regions, wherein adjacent regions have overlapping parts.

5. The visual assistance method according to claim 1, characterized in that: The coordinate system of the spatial coordinates takes the center of gravity of the person being assisted as the origin of the coordinates, and the coordinate system includes: a depth coordinate, a horizontal coordinate and a vertical coordinate; The direction in which the assisted person is facing in the horizontal direction is the positive direction of the depth coordinate, the direction in which the left hand side of the assisted person is located in the horizontal direction is the positive direction of the horizontal coordinate, and the vertically upward direction in the vertical direction is the positive direction of the vertical coordinate; The horizontal axis is divided into segments a, the vertical axis is divided into segments b, and the depth coordinate is divided into segments c, each segment having a corresponding natural language name; On the horizontal plane of the coordinate system, the area 180 degrees in front of the person being assisted is divided into d sectors, each sector has a corresponding natural language name; a, b, c, d are positive integers.

6. The visual assistance method according to claim 5, characterized in that: Generating the prompt content of the object prompt according to the first spatial coordinates includes: When the first spatial coordinate is located on the horizontal coordinate, the vertical coordinate or the depth coordinate, forming a description content about the first spatial coordinate in the prompt content according to the natural language name corresponding to the segment where the first spatial coordinate is located; When the first spatial coordinate changes, the description content of the first spatial coordinate is updated in real time according to the change of the first spatial coordinate.

7. The visual assistance method according to claim 1, characterized in that: The method further includes: determining the auxiliary information to be used according to the current visual assistance mode; The visual assistance mode includes any one or more of the following: walking mode, exploration mode, close-up mode and standby mode.

8. The visual assistance method according to claim 7, characterized in that: The determining of the auxiliary information to be used according to the current visual assistance mode includes: When the user is currently in the walking mode, the user is guided in the road direction according to the active guidance, and objects that need to be reminded detected by the object detection are screened out, and the objects that need to be reminded are described according to the object prompt; When the exploration mode is currently in progress, a spatial overall description is performed for the currently collected picture at a preset time interval; wherein the picture is divided into a plurality of grids, and the description content includes any one or more of the following: a scene description of each grid, a spatial coordinate description of each grid, a description of the average distance between each grid and the assisted person, and a description of the object information of the objects contained in each grid; When the close-up mode is currently in place, the object prompt is executed, wherein when the number of objects of the same type detected is greater than or equal to a preset number threshold, the detected objects of the same type are described together in the prompt content of the object prompt, and when the number of objects of the same type detected is less than the number threshold, the detected objects of the same type are described separately in the prompt content of the object prompt; and the prompt content only includes a description of the horizontal coordinate and / or the vertical coordinate, and does not include a description of the depth coordinate; When the current mode is in the standby mode, only the current visual assistance mode is uploaded as the standby mode, so that the calculation is stopped according to the standby mode.

9. The visual assistance method according to claim 1, characterized in that: The method further comprises: Collecting the input assistance help instruction, and determining the assistance information for the person being assisted according to the assistance help instruction; The auxiliary help instruction is a signal used to indicate that the person being assisted needs help; the auxiliary information corresponding to the auxiliary help instruction includes: sending help information.

10. A visual assistance system, characterized in that: include: Image data acquisition module, core calculation module, auxiliary information determination module and speech processing module; The image data acquisition module is configured to acquire image data of the environment; The image data includes: pictures and / or videos; The core computing module is configured to perform core computing based on the image data; the core computing includes any one or more of the following: distance detection, object detection, location detection, and scene description generation; The auxiliary information determination module is configured to determine auxiliary information for the assisted person according to the calculation result of the core calculation; The speech processing module is configured to convert the auxiliary information into interactive sentences, and transmit the interactive sentences to the assisted person in the form of speech; The auxiliary information includes any one or more of the following: distance alarm, object prompt, location change prompt, scene description and active guidance; Wherein, the distance alarm is used to issue a reminder to the assisted person when the distance between the assisted person and the detected object is less than or equal to a preset distance threshold; The object prompt is used to describe the object information corresponding to the detected object to the assisted person; the object information includes: name and / or spatial coordinates; The location change prompt is used to prompt the assisted person that the current location has changed and provide the assisted person with multiple possible locations; The scene description is used to describe the dynamic scene in the captured video in words; The active guidance is used to determine obstacles based on the object depth information obtained in the distance detection, and provide road direction guidance to the assisted person in the form of voice; The determining of the auxiliary information for the assisted person according to the calculation result of the core calculation includes: Determining the depth information detected by the distance detection as a first distance between an arbitrary first object on the picture and the assisted person, comparing the first distance with the distance threshold, and generating the distance alarm when the first distance is less than or equal to the distance threshold; Determine a first name of the first object according to the type of object detected in the object detection; determine a first height of the first object according to the object boundary information detected in the object detection; determine whether the first object can be suspended in the air according to the first name; determine a first spatial coordinate of the first object according to the first distance, the first height, and a first position of the first object relative to the assisted person detected in the object detection; correct the first spatial coordinate of the first object according to the information of whether the first object can be suspended in the air; generate a prompt content of the object prompt according to the first name, the corrected first spatial coordinate, and the suspended state of the first object; Retrieving n possible locations predicted last time according to the location detection, and comparing first location information obtained during the current location detection process with second location information of the n possible locations predicted last time, and when a difference rate between the first location information and the second location information is greater than or equal to a preset difference rate threshold, determining that the current location has changed, and re-predicting n possible locations according to the first location information; generating prompt content of the location change prompt according to the information of the change in the current location and the re-predicted n possible locations; Retrieving the voice description information obtained in the scene description generation calculation, and using the voice description information as the description content of the scene description; determining, based on the depth information detected in the distance detection, whether there is a continuous segment around the assisted person whose depth is greater than or equal to a preset depth threshold, and if the continuous segment exists, determining the continuous segment as a feasible route as the guidance direction of the active guidance; The step of correcting the first spatial coordinate of the first object according to the information of whether the first object can be suspended in the air comprises: The vertical coordinate in the object information of the first object and the depth information of the corresponding position of the first object are used to correct the vertical coordinate using a perspective algorithm; wherein, for an object classified as not being suspended in the air, its vertical coordinate is corrected to empty.

11. The visual assistance system according to claim 10, characterized in that: The core computing module includes: a distance detection module, an object detection module, a location detection module and a scene description generation module; the speech processing module includes a speech synthesis module and a speech interaction module; Among them, the distance detection module, the object detection module and the location detection module are set on the cloud server, the scene description generation module, the auxiliary information determination module and the speech synthesis module are set on the edge server, and the speech interaction module is set on the mobile terminal.

12. The visual assistance system according to claim 10, characterized in that: It also includes: an input collection module and a help module; The input collection module is configured to collect input assistance help instructions; the assistance help instructions are signals used to indicate that the person being assisted needs help; The help-seeking module is configured to determine the auxiliary information for the person being assisted according to the auxiliary help-seeking instruction; the auxiliary information corresponding to the auxiliary help-seeking instruction includes: sending help-seeking information.

13. A visual aid device, comprising a processor and a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, characterized in that: When the instructions are executed by the processor, the visual assistance method as described in any one of claims 1 to 9 is implemented.

Images