Method and device for assisting blind people in shopping

Abstract

This application discloses a method and device for assisting blind people in shopping. The blind person's wearable device is equipped with a camera module. The method includes: determining, based on the camera module, whether the image perceived by the camera module contains complete object frame information; if the image does not contain complete object frame information, and if the blind person has not yet decided to purchase the product, generating a reminder indicating that the blind person is too close to the product, prompting the blind person to adjust their position in response to the reminder, thereby confirming the complete product information. This application enables blind people to independently select their favorite products in shopping venues.

Description

Technical Field

[0001] The present application relates to the field of shopping guide technology, and in particular to a method and device for assisting blind people in shopping. Background Art

[0002] Currently, the primary solution for blind people shopping in large supermarkets is to have them accompanied by family members, or to have shopping guides and service staff guide them through the entire shopping process. However, the current approach for blind people to enjoy an immersive shopping experience in supermarkets is still imperfect, has many shortcomings, is subject to significant external influences, and is difficult to implement. Summary of the Invention

[0003] The present application provides a method and device for assisting blind people in shopping, which can enable blind people to choose their favorite products by themselves in shopping places.

[0004] To achieve the above-mentioned objectives, the present application provides a method for assisting a blind person in shopping. The blind person's wearable device is provided with a camera module. The method comprises:

[0005] Determine, based on the camera module, whether the image perceived by the camera module has complete object border information;

[0006] If there is no complete object bounding box information and the blind person has not yet decided to purchase the product, a reminder is generated that the person is too close to the product, so that the blind person can adjust their position based on the reminder and thus determine the complete information of the product.

[0007] The method further includes:

[0008] If there is no complete object bounding box information, a local area detection algorithm is used to detect the area of ​​interest of the product, and feature extraction and comparison are performed on the area of ​​interest to determine the product information.

[0009] The steps of detecting the area of ​​interest of the product using the local area detection algorithm include:

[0010] The Harris-Laplace algorithm is used to extract corner points in the product image captured by the camera module, and the extracted corner points are clustered to determine the product's region of interest.

[0011] The step of generating a reminder of being too close to the product includes:

[0012] When the blind person adjusts his position to a position where the camera module perceives an image with complete object frame information based on the reminder, optical character recognition is performed on the product image within each object frame of the camera module perceives the image to determine the complete information of the product.

[0013] The method further includes:

[0014] Play the information of the determined product through the voice device;

[0015] Determine the product that the blind person decides to buy based on the blind person's voice collected by the voice device;

[0016] The camera module determines the relative position between the blind person's hand and the product they want to buy, and uses voice equipment to dynamically prompt the blind person based on the relative position so that the blind person can accurately pick up the product they want to buy.

[0017] Wherein, the wearable device is a glove, and the camera module is used to determine whether the image perceived by the camera module has complete object border information, including:

[0018] Based on the camera module on the glove, it is determined in real time whether the image perceived by the camera module has complete object border information;

[0019] If there is complete object frame information, optical character recognition is performed on the product image within each object frame of the image perceived by the camera module to determine the complete information of each product in the image perceived by the camera module, and the information of all products in the image perceived by the camera module is played through the voice device.

[0020] The step of determining whether the image perceived by the camera module has complete object frame information based on the camera module includes:

[0021] Confirm information about the goods that the blind person intends to purchase through voice equipment;

[0022] Determine the location of the intended purchase based on the intended purchase information and automatically plan a route based on the blind person's current location;

[0023] Navigation for the blind based on planned paths.

[0024] To achieve the above-mentioned purpose, the present application provides a device for assisting blind people in shopping, which includes a detection module and a reminder module:

[0025] The detection module is used to determine whether the image perceived by the camera module has complete object frame information based on the camera module;

[0026] The reminder module is used to generate a reminder that the blind person is too close to the product if there is no complete object border information and the blind person has not decided to buy the product, so that the blind person can adjust his position based on the reminder and thus determine the complete information of the product.

[0027] To achieve the above objectives, the present application provides a system for assisting blind people in shopping, which includes a processor; the processor is used to execute instructions to implement the above method.

[0028] To achieve the above objectives, the present application provides a computer-readable storage medium, which stores a program file capable of implementing the above method.

[0029] This application first determines whether the image perceived by the camera module on the blind person's wearable device has complete object frame information. If it is confirmed that the image perceived by the camera module does not have complete object frame information, and the blind person has not decided to buy the product, a reminder is generated that the distance to the product is too close, so that the blind person can adjust the camera module to a position where the complete product can be captured based on the reminder, so as to obtain complete information about the product. In this way, the blind person can make a conclusion on whether to buy the product based on the complete information of the product, which makes it easier for the blind person to choose their favorite products. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0031] Figure 1 This is a flowchart of an implementation method of the present invention for assisting blind people in shopping;

[0032] Figure 2 This is a schematic diagram of an embodiment of a wearable device for the blind in the method of assisting the blind in shopping in the present application;

[0033] Figure 3 This is a schematic diagram of perception image detection according to an embodiment of the method for assisting blind people in shopping;

[0034] Figure 4 This is a schematic diagram of perception image detection in another embodiment of the method for assisting blind people in shopping;

[0035] Figure 5 This is a flow chart of another embodiment of the method for assisting blind people in shopping;

[0036] Figure 6 This is a schematic diagram of perception image detection according to another embodiment of the method for assisting blind people in shopping;

[0037] Figure 7 This is a schematic diagram of perception image detection according to another embodiment of the method for assisting blind people in shopping;

[0038] Figure 8 This is a structural diagram of an embodiment of the system for assisting blind people in shopping;

[0039] Figure 9 This is a schematic diagram of the structure of the positioning device in the blind shopping assistance system of the present application;

[0040] Figure 10 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0041] The principle of the present application is described and illustrated. It will be appreciated that those skilled in the art will be able to design various arrangements, and although arrangements are not explicitly described or shown herein, they embody the principle of the present application and are included within the scope of the present application. In addition, all examples herein are primarily and explicitly intended to be used for teaching purposes, to assist the reader in understanding the principle of the present application and the concept provided by the inventor, thereby deepening the field, and all examples should not be interpreted as being limited to the examples and conditions of such specific elaboration. In addition, unless otherwise indicated (for example, "or in addition" or "or in an alternative"), the term "or" as used herein refers to non-exclusive "or" (that is, "and / or"). Furthermore, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments.

[0042] Please refer to the following for details: Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for assisting blind people in shopping. It should be noted that the following numbers are only used to simplify the description and are not intended to limit the order of execution of the steps. The steps of this embodiment can be executed in any order without violating the technical concept of this application.

[0043] S101: Determine, based on the camera module, whether the image perceived by the camera module has complete object frame information.

[0044] It is possible to first determine whether the image perceived by the camera module on the blind person's wearable device has complete object frame information. If it is confirmed that the image perceived by the camera module does not have complete object frame information, and the blind person has not decided to buy the product, a reminder is generated that the distance to the product is too close, so that the blind person can adjust the camera module to a position where the complete product can be photographed based on the reminder, thereby obtaining complete information about the product. The blind person can then conclude whether the product needs to be purchased based on the complete information about the product, making it easier for the blind person to choose their favorite products.

[0045] Among them, a camera module is provided in the wearable device of the blind. The wearable device can be a device worn on the upper body of the blind, such as a necklace, a watch, a glove, etc. More preferably, the camera module can be located on the blind's glove, so that the camera module moves synchronously with the blind's hand, and the image captured by the camera module can guide the blind's hand to move away from the product to obtain complete information about the product, or guide the blind to move to other products to obtain information about other products, so that the blind can compare multiple products to confirm the product they decide to buy, or guide the blind's hand to get close to the product they decide to buy so that the blind can get the product they decide to buy. Further, if Figure 2 As shown, the camera module can be located in the palm of the blind person's glove. In addition, the type of the camera module is not limited, for example, it can be an AI perception camera.

[0046] Optionally, in step S101, an object detection algorithm such as YOLO can be used to detect the image sensed by the camera module; if Figure 3 As shown, at least one object can be detected from the perception image of the camera module, and at least one complete object frame is presented on the perception image, then it can be confirmed that the perception image of the camera module has complete object frame information; if Figure 4 As shown, if no complete object boundary can be detected in the image perceived by the camera module, it is determined that the image perceived by the camera module does not have complete object frame information, and the process goes to step S102.

[0047] S102: If there is no complete object border information and the blind person has not yet decided to purchase the product, a reminder is generated indicating that the blind person is too close to the product.

[0048] When confirming that the camera module senses an image without complete object border information and the blind person has not yet decided to purchase the product, a reminder that the blind person is too close to the product can be generated so that the blind person can adjust their position based on the reminder, thereby facilitating the determination of complete product information.

[0049] Optionally, if the blind person does not confirm the final purchase of the product, or if the blind person's hand is not sensed to be close to the product after the information of the product in the image sensed by the camera module is informed to the blind person, it is confirmed that the blind person has not decided to buy the product.

[0050] After generating a reminder that the distance to the product is too close, the reminder can be played out through a voice device so that the blind person can know it.

[0051] It is understood that there are many ways to remind users that they are too close to a product. For example, a voice message could be played directly stating "You are too close to the product." Another example is a voice message with a different voice format than a message with complete border information. This allows a blind person to easily understand that their wearable device equipped with a camera module is too close to a product. For example, a message with complete border information could be "You are currently browsing X items, namely...", while a message without complete border information could be "You are currently browsing..."

[0052] In this embodiment, it is determined whether the image perceived by the camera module on the blind person's wearable device has complete object frame information. If it is confirmed that the image perceived by the camera module does not have complete object frame information, and the blind person has not decided to buy the product, a reminder is generated that the distance to the product is too close, so that the blind person can adjust the camera module to a position where the complete product can be photographed based on the reminder, so as to obtain complete information about the product. In this way, the blind person can make a conclusion on whether to buy the product based on the complete information of the product, which is convenient for the blind person to choose the product they like.

[0053] See also Figure 5 , Figure 5 This is a flow chart of another embodiment of the method for assisting blind people in shopping. It should be noted that the following numbers are only used to simplify the description and are not intended to limit the order of execution of the steps. The steps of this embodiment can be executed in any order without violating the technical concept of this application.

[0054] S201: Determine the product that the blind person intends to purchase through a voice device.

[0055] The blind person can first confirm the goods he / she intends to purchase through a voice device, so that a route can be planned based on the goods he / she intends to purchase, so as to guide the blind person to the goods he / she intends to purchase, so that the blind person can reach the area of ​​the goods he / she wants to purchase independently.

[0056] The voice device can be worn by the blind person and can capture relatively clear speech. Voice recognition can then be performed to determine the items the blind person intends to purchase. Alternatively, the voice device worn by the blind person can be distributed upon entering a shopping mall. Alternatively, the voice device worn by the blind person can be their own, requiring the blind person to establish a communication connection between the voice device and the mall's system under the guidance of a staff member upon entering the mall.

[0057] In other embodiments, multiple voice devices are evenly arranged in the shopping mall, and each voice device collects voice information around it. Based on the voice information collected by each voice device, the speaker of each voice information and the speaker's speech content can be determined. Therefore, as long as the blind person speaks the information of the goods he intends to buy, the system can confirm the goods he intends to buy based on the multiple voice devices in the shopping mall.

[0058] S202: Automatically plan a route based on the goods to be purchased.

[0059] After confirming the items that the blind person intends to purchase, the route can be automatically planned based on the items to be purchased.

[0060] In one implementation, the system stores a map that shows the correspondence between product types and locations. The system can confirm the shelf number and location of the product based on the type of product the blind person intends to purchase, and automatically plan a route based on the blind person's current location.

[0061] In another implementation, the system stores a map that corresponds to product names and locations. The system can confirm the location of the product that the blind person intends to purchase based on the name of the product, and automatically plan a path based on the blind person's current location, so that the blind person can be guided to the product to be purchased based on the planned path.

[0062] Optionally, in step S202, a specific path planning detail scheme may be as follows.

[0063] In one implementation, the crowd density of each channel in the shopping mall may be obtained first, and then path planning may be performed based on the crowd density of each channel.

[0064] For example, path planning can be performed based on the principle of minimizing crowd density along the path. Specifically, all initial paths from the blind person's current location to the goods the blind person intends to purchase can be determined first; then, a path with the minimum crowd density can be selected from all the initial paths as the final path for navigating the blind person.

[0065] For another example, all initial paths between the blind person's current location and the item they intend to purchase can be eliminated if they contain a crowd density greater than a threshold. Then, from all remaining initial paths, a path with the lowest crowd density is selected as the final navigation path for the blind person. The density threshold can be set based on actual circumstances and is not limited here; for example, it can be 3 people / m. The minimum crowd density mentioned above can refer to the minimum average crowd density.

[0066] For another example, path planning can be performed based on the principle of minimizing crowd density and minimizing the total path. Specifically, in step S202, at least one iteration is performed based on the crowd density of each channel. In the k+1th iteration, based on the planning starting point and the basic path determined in the kth iteration, the channel cluster with the lowest crowd density among all channel clusters containing the planning starting point is determined. From the channel cluster with the lowest crowd density, the channel with the shortest distance to the blind person's location and containing the planning starting point is selected. If the channel with the shortest distance and containing the planning starting point meets the preset conditions, the basic path is updated based on the channel with the shortest distance and containing the planning starting point, and the planning starting point is updated to the end point of the channel. The preset conditions can be understood as: the total distance of the replacement path obtained based on the channel with the shortest distance and containing the planning starting point is less than the total distance of the basic path. Prior to this, a clustering scheme such as an improved k-means algorithm can be used to determine the blind person's current location and the channel they intend to purchase as the initial central channel. Then, based on the principle of closest center point, the remaining channels are assigned to the checkpoint cluster represented by the initial central channel, thereby generating at least two channel clusters. The crowd density of each channel cluster is then determined based on the crowd density of at least some channels in each channel cluster. For example, the crowd density of the center point of each channel cluster (which can be the initial central channel and final central channel of each channel cluster) can be used as the crowd density of each channel cluster.

[0067] Alternatively, a camera in the mall (e.g., an overhead camera) can be used to capture crowd sampling images of each channel between the blind person's current location and the item they intend to purchase. The crowd density in each channel can then be determined based on the crowd sampling images. Alternatively, a simple path planning method can be used to determine all paths between the blind person's current location and the item they intend to purchase, thereby obtaining all initial paths.

[0068] In another implementation, path planning can be performed based on the blind person's current heading. Specifically, all initial paths between the blind person's current location and the items they intend to purchase can be determined. Paths whose initial walking direction does not match their current heading are then eliminated from these initial paths to obtain a set of remaining paths. A final path for navigation is then determined from this set of remaining paths based on criteria such as crowd density, number of turns, or total distance traveled. This prevents invalidating the planned path by directly using the blind person's current heading as the initial walking direction if the blind person's current heading does not match their initial walking direction. For example, assuming the blind person's current heading is facing south, and the determined initial paths include paths that require a right turn, paths that require going straight ahead, and paths that require turning backwards, the paths that require a right turn and paths that require turning backwards are eliminated, and the final path is selected from all paths that require going straight ahead. The blind person's current heading can be determined based on an image of the blind person captured by a camera within the mall.

[0069] In another implementation, a path with the fewest turns can be selected from all initial paths between the blind person's current location and the goods the blind person intends to purchase to determine a final path for navigating the blind person. In this way, the blind person does not need to make too many turns while walking, which can reduce the probability of the blind person taking the wrong path.

[0070] In another implementation, the path with the most products of the same type as the products the blind person intends to purchase can be selected from all initial paths between the blind person's current location and the products the blind person intends to purchase to determine the final path for navigation for the blind person.

[0071] In another implementation, path planning can be performed based on the blind person's body shape. Specifically, the blind person's body width can be first determined. Then, from all initial paths between the blind person's current location and the item they intend to purchase, any paths where the difference between the width of the channel and the blind person's identity width is less than a width threshold are eliminated. A final navigation path for the blind person is then selected from all remaining initial paths to avoid narrow channels in the final planned path that would make it difficult for the blind person to navigate, thereby improving the shopping experience for the blind person. The width threshold can be set based on actual conditions and is not a limitation here; for example, it can be 50 cm.

[0072] In another implementation, path planning can be performed based on the presence of obstacles within the passage. Specifically, paths with a high number of obstacles can be eliminated from all initial paths between the blind person's current location and the item they intend to purchase. A final path for navigation for the blind person can then be selected from all remaining initial paths.

[0073] Furthermore, if the blind person's current location and the item they intend to purchase are not on the same floor, the elevator route is prioritized over the escalator route, which in turn is prioritized over the stairs. This means that when planning a route, if there is an elevator in the mall, the system will try to plan a route that uses the elevator to change floors. If there is an escalator but no elevator, the system will try to plan a route that uses the escalator to change floors.

[0074] S203: Navigating for the blind person based on the planned path.

[0075] After automatically planning a path based on the goods that the blind person intends to purchase, navigation can be performed for the blind person based on the planned path to guide the blind person to move forward to the goods that he intends to purchase.

[0076] Optionally, when a blind person's position deviates during movement, the system will automatically and promptly give a voice reminder to inform the blind person that the current path is wrong. At the same time, it will automatically plan a new path and guide the blind person to move along the new path until the blind person moves to the shelf where the intended product is located. Real-time reminders will be provided during the navigation process for the blind to prevent the blind person from going in the wrong direction and improve their shopping experience.

[0077] Optionally, in the process of navigating for the blind, it is possible to determine whether the blind's walking path is wrong based on the blind's precise positioning, so as to avoid the inability to correctly navigate the blind due to inaccurate positioning of the blind. The precise positioning of the blind can be determined based on the cameras in the shopping mall. Alternatively, the precise positioning of the blind can be determined based on the position information read by the sensing module on the blind from the sensing chip near the blind. Specifically, a plurality of sensing chips storing position information are provided at each channel in the shopping mall (for example, on the ground or on the shelves of the channel); when the blind walks to the channel, the sensing module on the blind can read the position information stored in at least one sensing chip of the channel, and then the positioning system determines the precise positioning of the blind based on the position information read by the sensing module.

[0078] Optionally, the system can provide navigation for the blind through voice devices. Specifically, the system can transmit real-time location updates to the blind through the system's voice output interface and headphones, such as: "Please continue straight for ten meters, nine meters, etc."; "Please notify me in advance when you encounter a turn, such as: "Turn right in five meters ahead."

[0079] Optionally, during navigation for the blind, it is also possible to confirm whether there is an obstacle in front of the blind through a camera device in the mall; when there is an obstacle in front of the blind, the type, size and / or location information of the obstacle can be informed to the blind so that the blind can cross or avoid the obstacle. If the obstacle is a person, the blind can also communicate with the obstacle so that the person in front of the blind can give way to avoid a collision between the blind and the person in front.

[0080] In addition, during the navigation process for the blind, the system can also introduce information such as the types of goods on the shelves near the blind through voice equipment. In this way, when there are items that the blind want to buy on the navigation path, the blind can stop and select the items without having to return after reaching the intended purchase. This improves the shopping efficiency of the blind and allows the blind to experience the fun of immersive shopping.

[0081] When a blind person reaches the location of the product he intends to purchase, the system automatically prompts that he has arrived and informs the blind person of the location of the product he intends to purchase, for example: the shelf on the right of the product.

[0082] S204: Determine, based on the camera module, whether the image perceived by the camera module has complete object frame information.

[0083] After guiding the blind person to the shelf where they plan to purchase the goods, the system interacts with the camera module to confirm whether the image perceived by the camera module has complete object border information; if it is confirmed that the image perceived by the camera module has complete object border information, the process goes to step S206; if it is confirmed that the image perceived by the camera module does not have complete object border information, the process goes to step S205.

[0084] Among them, the step of the system interacting with the camera module to confirm whether the camera module perceives an image with complete object border information may include: the system issues a work instruction to the camera module, and the camera module confirms on its own whether the perceived image has complete object border information.

[0085] Among them, the system interacts with the camera module to confirm whether the camera module perceives the image with complete object border information, or it can be: the system obtains the image taken by the camera module and detects the image to confirm whether the image has complete object border information.

[0086] Optionally, when a blind person uses the camera module on a wearable device to perceive a product, the camera module captures the product in the area where the blind person's palm slides in real time and performs real-time target detection and recognition. During the detection process, algorithms such as YOLO can be used to detect and obtain rough category information of the product. The camera module also determines whether the image perceived by the current camera module has complete object bounding box information. Based on the confirmation result of whether the object bounding box information is complete, the camera module's perceived image can be detected and recognized using different detection algorithms to obtain information about the product in the image perceived by the camera module. Furthermore, the camera module compares the captured image content with the extracted product features and price information stored in a database, and transmits the results to the blind person in real time through the system's voice output interface.

[0087] In addition, since the blind person may want to obtain information about other products before reaching the location of the product he intends to purchase, during the navigation process, the system can turn on the shooting and recognition functions of the camera module on the wearable device when the blind person wants to obtain product information, so that the blind person can obtain information about the product pointed to by his wearable device.

[0088] Specifically, the cameras in the shopping mall can be used to detect the body shape of the blind person; to determine whether the body shape of the blind person matches the body shape of a person using a wearable device to detect product information; if it matches, it can be determined that the blind person wants to obtain product information. At this time, the shooting and recognition functions of the camera module on the wearable device can be turned on, so that when the blind person wants to obtain information, the camera module can be automatically turned on to obtain product information without the blind person having to issue instructions himself, allowing the blind person to experience the fun of immersive shopping.

[0089] Alternatively, in other embodiments, the shooting and recognition instructions of the camera module of the wearable device can be activated in response to the instruction to obtain product information issued by the blind person.

[0090] In addition, when no product information can be identified in the image captured by the camera module of the wearable device, a prompt can be given to the blind person: there are no products in the current area, please move the wearable device so that the blind person can quickly find the products he wants to buy.

[0091] S205: Detecting the perceived image of the camera module using a local area detection algorithm to determine information about the commodity in the perceived image.

[0092] After confirming that the image perceived by the camera module does not have complete object border information, a local area detection algorithm can be used to detect the image perceived by the camera module to detect the area of ​​interest of the product in the perceived image, and feature extraction and comparison can be performed on the area of ​​interest to determine the information of the product in the perceived image, so as to use the local area detection algorithm to determine at least partial information of the product for reference by the blind, so that the blind can confirm whether they need to obtain complete information about the product.

[0093] Optionally, the Harris-Laplace algorithm can be used to extract the region of interest of the product in the perception image captured by the camera module. Among them, the Harris-Laplace interest point extraction algorithm combines the Harris algorithm and the Laplace scale space to achieve scale invariance. The Harris-Laplace interest point extraction algorithm is derived and calculated based on the Harris operator. The Harris-Laplace algorithm essentially uses the Harris corner detection method under multiple scales. The optimal scale is represented by the scale at which a discriminant function reaches the maximum value under the local structure, and the size of the region of interest is determined by the Laplace operator. The feature extraction operator formula of the Harris-Laplace operator for the product region of interest is:

[0094] |σ 2 (I xx (X,σ))+L yy (X,σ)|

[0095] Where σ is the scale of the Laplace formula, I xx and L yy It is the second-order derivative of the image signal, from which the edge features and Harris-Laplace interest point features on the image can be obtained.

[0096] The Harris corner detection algorithm uses a local window to move across the image to determine whether the grayscale changes significantly. If the grayscale values ​​within the window (on the gradient map) change significantly, then a corner exists in the area where the window is located. The basic principle is as follows:

[0097]

[0098] Specifically, the real symmetric matrix is ​​diagonalized, where R is the rotation factor, which has no effect on the change components in the two orthogonal directions; λ1 and λ2 represent the eigenvalues ​​extracted from the changes in the two orthogonal directions. The eigenvalue represents the change of the pixel in the two directions. The larger the eigenvalue, the greater the change in that direction. When both eigenvalues ​​exceed the threshold, the point is a corner point.

[0099] Furthermore, the image captured by the camera module can be converted to grayscale and then the region of interest can be extracted using the Harris-Laplace detection algorithm. Figure 6 Harris-Laplace corner feature extraction and comparison are performed on the area of ​​interest shown in the white box in the main view of the paper drawing. Figure 7Harris-Laplace corner feature extraction and comparison are performed on the area of ​​interest shown in the white box in the top view of the paper drawing.

[0100] Optionally, the features of the region of interest detected in the image captured by the camera module are compared with a previously collected library of features of local regions of interest of the product to obtain information about the product in the image perceived by the camera module, and the product information is played out through a voice device, such as "You are currently browsing..." The blind person can move the wearable device with the camera module backward as needed to obtain complete information about the product, including price and other information, so that the blind person can decide which product to buy.

[0101] Furthermore, if the image perceived by the camera module lacks complete object frame information, and the blind person has not yet decided to purchase the product, a reminder can be generated indicating that the person is too close to the product, prompting the blind person to adjust their position in response to the reminder, thereby determining complete information such as the product name, production date, shelf life, and / or substrate price. Furthermore, if the blind person adjusts their position to a point where the image perceived by the camera module contains complete object frame information, optical character recognition (OCR) can be performed on the product image within each object frame in the image perceived by the camera module to determine the complete product information, allowing the product information to be quickly read using optical character recognition.

[0102] S206: Perform optical character recognition on the product image within each object frame of the image perceived by the camera module to determine complete information about the product.

[0103] If the image sensed by the camera module has complete object frame information, optical character recognition is performed on the product image within each object frame of the image sensed by the camera module to determine the complete information of the product.

[0104] S207: Play the information of the determined product through the voice device.

[0105] After determining that the camera module perceives the information of the product in the image based on steps S205 and S206, the information of the product can be integrated into voice information and played out through a voice device to let the blind know the information of the product in the image perceived by the camera module, thereby facilitating the blind to decide on the product to purchase.

[0106] S208: Determine the product that the blind person decides to purchase based on the blind person's voice collected by the voice device.

[0107] After the information of the determined product is played through the voice device, the product that the blind person decides to buy can be determined, so that the blind person can be dynamically prompted through the relative position between the blind person's hand and the product decided to be purchased determined based on the camera module, so that the blind person can dynamically adjust the hand position and get the product he decided to buy.

[0108] In one implementation, a blind person can decide which product he or she wants to buy based on the product information captured and transmitted by the camera module. After making a decision, he or she can simply touch and pick it up with his or her hands. When the camera module senses that the blind person's hand is getting closer to a certain product, the product can be regarded as the product that the blind person has decided to buy.

[0109] In another implementation, a blind person can decide which product he wants to buy based on the product information captured and transmitted by the camera module. The blind person can say the name of the product he decides to buy and other information. In this way, the name of the product that the blind person finally decides to buy and other information can be obtained through the voice device, thereby knowing the product that the blind person decides to buy.

[0110] S209: Determine the relative position between the blind person's hand and the product to be purchased based on the camera module, and dynamically prompt the blind person based on the relative position until the blind person gets the product to be purchased.

[0111] After determining the product that the blind person has decided to buy based on step S208, the relative position between the blind person's hand and the product that the blind person has decided to buy can be determined based on the camera module, and then the voice device can be used to dynamically prompt the blind person based on the relative position so that the blind person can accurately pick up the product that he or she has finally decided to buy.

[0112] Optionally, after the blind person purchases the goods through the above steps, the blind person can inform the system of the information of the next item he or she intends to purchase. At this time, the above method of assisting the blind person in shopping will be executed to allow the blind person to finally determine the goods to be purchased based on the information of the next item he or she intends to purchase and the information of at least one item on the shelf, and pick up the goods to be purchased.

[0113] Alternatively, after the blind person purchases the goods through the above steps, he or she can tell the system that he or she wants to check out and leave. At this time, the system will automatically plan a path to guide the blind person to the checkout, or the system will automatically summarize and settle the prices of all items purchased by the blind person; and then guide the blind person out of the shopping area.

[0114] Based on the above-mentioned method of assisting blind people in shopping, it is basically possible for blind people to shop independently in large supermarkets and enjoy the fun of immersive shopping.

[0115] The above-mentioned method of assisting blind people in shopping requires relatively little investment in hardware equipment in the early stage, has high reusability, and has the advantage of "development in one place, use in multiple places". It not only avoids the disadvantage that items in shopping malls can only be placed in an agreed order, but also makes it possible for blind people to shop independently in large supermarkets. It also avoids the disadvantage that some customers mistakenly put back certain items, resulting in the blind getting what they don't want. In addition, for supermarkets that want to change their business strategies and adjust the placement of goods, the energy and financial resources consumed are relatively small. They only need to re-update the product location information in the navigation and positioning system according to the new product placement order. Moreover, gloves equipped with AI perception cameras can help blind people choose their favorite types of goods in shopping malls and supermarkets and experience the fun of immersive shopping.

[0116] The present application also provides a device for assisting blind people in shopping, which further includes a detection module and a reminder module.

[0117] The detection module determines whether the image perceived by the camera module has complete object frame information based on the camera module.

[0118] The reminder module is used to generate a reminder that the blind person is too close to the product if there is no complete object frame information and the blind person has not decided to buy the product, so that the blind person can adjust his position based on the reminder, thereby facilitating the determination of the complete information of the product.

[0119] Optionally, if there is no complete object bounding box information, the detection module is used to detect the region of interest of the product using a local region detection algorithm, and perform feature extraction and comparison on the region of interest to determine the information of the product.

[0120] Optionally, the detection module is configured to extract corner points in the product image captured by the camera module using a Harris-Laplace algorithm, and perform clustering connection on the extracted corner points to determine a region of interest of the product.

[0121] Optionally, the detection module is used to perform optical character recognition on the product image within each object frame of the image perceived by the camera module when the blind person adjusts his position to a position where the camera module perceives the image with complete object frame information based on the reminder, so as to determine the complete information of the product.

[0122] Optionally, the device for assisting the blind in shopping also includes a guidance module, which is used to play the information of the determined goods through a voice device; determine the goods that the blind person finally purchases based on the blind person's voice collected by the voice device; determine the relative position between the blind person's hand and the goods finally purchased based on the camera module, and use the voice device to dynamically prompt the blind person based on the relative position so that the blind person can accurately pick up the goods finally purchased.

[0123] Among them, the wearable device is a glove, and the detection module is used to determine in real time whether the image perceived by the camera module has complete object border information based on the camera module on the glove; if there is complete object border information, optical character recognition is performed on the product image within each object border of the image perceived by the camera module to determine the complete information of each product in the image perceived by the camera module, and the information of all products in the image perceived by the camera module is played through the voice device.

[0124] Among them, the guidance module is used to determine the information of the goods that the blind person intends to purchase through a voice device; determine the location information of the goods to be purchased based on the information of the goods to be purchased, and automatically plan a path based on the current location of the blind person; and navigate for the blind person based on the planned path.

[0125] See 8, Figure 8 1 is a schematic diagram of an embodiment of a blind person shopping assistance system 10 of the present application. The blind person shopping assistance system 10 of the present application includes a processor 12. The processor 12 is configured to execute instructions to implement the above-mentioned blind person shopping assistance method.

[0126] The processor 12 may also be referred to as a CPU (Central Processing Unit). The processor 12 may be an integrated circuit chip having signal processing capabilities. The processor 12 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor 12 may be any conventional processor.

[0127] Optionally, the blind shopping assistance system 10 includes: Figure 9 The processor 12 is disposed in the positioning device, and the positioning device can execute the above-mentioned method of assisting blind people in shopping.

[0128] Optionally, the shopping assistance system for the blind 10 may further include a voice device for playing navigation information and product information output by the positioning device.

[0129] Accordingly, if Figure 9 As shown, the positioning device may include an audio input and output interface for connecting to a voice device (such as headphones or speakers, etc.) on the blind person, so as to transmit navigation information or product information to the voice device through the audio input and output interface so as to be played out through the voice device.

[0130] In addition, the assisted blind shopping system 10 may also include a camera module located on the wearable device, which is used to collect perceived images and can be used to detect and identify the perceived images to determine information about the products in the perceived images.

[0131] Alternatively, as Figure 9 As shown, the positioning device may also include a camera module input and output interface for connecting to a camera module of a device worn by the blind person, so as to obtain the perceived image captured by the camera module or the product information obtained by the camera module from identifying the perceived image, so as to determine the information of the product in the perceived image, and transmit the information of the product in the perceived image to the voice device through the camera module input and output interface, so that the voice device can play it, so that the blind person can determine the final product to purchase by himself based on the product information heard.

[0132] The positioning device may further include a memory 11 for storing instructions and data required for the processor 12 to operate.

[0133] See also Figure 10 , Figure 10 Schematic diagram of the structure of the computer-readable storage medium in the embodiment of the present application. The computer-readable storage medium 20 of the embodiment of the present application stores instruction / program data 21, which, when executed, implements the method provided by any embodiment of the above-mentioned method of the present application and any non-conflicting combination. Among them, the instruction / program data 21 can form a program file and be stored in the above-mentioned storage medium 20 in the form of a software product, so that a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) executes all or part of the steps of the various embodiments of the present application. The aforementioned storage medium 20 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer, server, mobile phone, tablet and other devices.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0135] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0136] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0137] The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for assisting blind people in shopping, characterized in that: The wearable device for the blind is provided with a camera module, and the method includes: Determining, based on the camera module, whether the image perceived by the camera module has complete object frame information; If there is no complete object frame information and the blind person has not yet decided to purchase the product, a reminder is generated indicating that the person is too close to the product, so that the blind person can adjust their position according to the reminder and thus determine the complete information of the product; The method further comprises: If there is no complete object bounding box information, a local area detection algorithm is used to detect the region of interest of the product, and feature extraction and comparison are performed on the region of interest to determine the information of the product; The wearable device is a glove, and determining whether the image sensed by the camera module contains complete object frame information based on the camera module includes: Based on the camera module on the glove, it is determined in real time whether the image perceived by the camera module has complete object frame information; If there is complete object frame information, optical character recognition is performed on the product image within each object frame of the image perceived by the camera module to determine the complete information of each product in the image perceived by the camera module, and the information of all products in the image perceived by the camera module is played through the voice device.

2. The method according to claim 1, characterized in that The step of detecting the region of interest of the commodity using a local region detection algorithm includes: The Harris-Laplace algorithm is used to extract corner points in the product image captured by the camera module, and the extracted corner points are clustered and connected to determine the region of interest of the product.

3. The method according to claim 2, characterized in that The step of extracting corner points from the product image captured by the camera module using the Harris-Laplace algorithm includes: The following Laplace feature extraction operator formula is used to extract edge features and corner features in the product image; ； Among them, the It's the scale. and is the second-order derivative of the product image.

4. The method according to claim 1, wherein The step of generating a reminder of being too close to the product includes: When the blind person adjusts his position to a position where the image perceived by the camera module has complete object frame information based on the reminder, optical character recognition is performed on the product image within each object frame of the image perceived by the camera module to determine the complete information of the product.

5. The method according to claim 4, characterized in that The method further comprises: Playing the determined information of the product through a voice device; Determining the commodity that the blind person decides to purchase based on the speech of the blind person collected by the speech device; The relative position between the blind person's hand and the product to be purchased is determined based on the camera module, and the voice device is used to dynamically prompt the blind person based on the relative position until the blind person picks up the product to be purchased.

6. The method according to claim 1, characterized in that The step of determining whether the image perceived by the camera module has complete object frame information based on the camera module includes: Determining information about the merchandise that the blind person intends to purchase through a voice device; Determining the location information of the commodity to be purchased based on the information of the commodity to be purchased, and automatically planning a route based on the current location of the blind person; Navigation is performed for the blind person based on the planned path.

7. The method according to claim 6, characterized in that The step of automatically planning a path based on the current location of the blind person includes: determining a current orientation of the blind person; Eliminate paths whose initial walking directions do not match the current direction of the blind person from all initial paths between the blind person's current location and the product to be purchased, to obtain a set of remaining paths; A final path for navigation for the blind is determined from the remaining path set based on the crowd density in the passages in each initial path, the presence of obstacles in the passages in each initial path, the number of turns and / or the total distance of each initial path.

8. The method according to claim 6, characterized in that The step of automatically planning a path based on the current location of the blind person includes: Obtaining the crowd density of each channel between the current location of the blind person and the product to be purchased; From all initial paths between the current location of the blind person and the commodity to be purchased, a path with the smallest crowd density is selected as a final path for navigating the blind person.

9. The method according to claim 8, characterized in that The step of selecting a path with the smallest crowd density from all initial paths between the current location of the blind person and the product to be purchased as a final path for navigating the blind person includes: Eliminate all paths from the initial paths where the crowd density is greater than the density threshold; A path with the smallest crowd density is selected from all remaining initial paths as the final path for navigation for the blind person.

10. The method according to claim 6, characterized in that The step of automatically planning a path based on the current location of the blind person includes: Determine the blind person's body width; Eliminate paths in which the difference between the channel width and the body width is less than a width threshold from all initial paths between the current location of the blind person and the product to be purchased; A final path for navigation for the blind person is selected from all remaining initial paths.

11. The method according to claim 6, characterized in that The step of navigating for the blind person based on the planned path includes: During navigation for the blind person, confirming whether there is an obstacle in front of the blind person through a camera device in the mall; When there is an obstacle in front of the blind person, the type information, size information and / or position information of the obstacle is prompted to the blind person.

12. A shopping aid for the blind, characterized in that: The device for assisting blind people in shopping is provided with a camera module, and the device comprises: A detection module, configured to determine, based on the camera module, whether the image perceived by the camera module has complete object frame information; a reminder module, configured to generate a reminder that the blind person is too close to the product if there is no complete object frame information and the blind person has not yet decided to purchase the product, so that the blind person can adjust their position according to the reminder to determine the complete information of the product; The reminder module is further configured to detect the region of interest of the product using a local region detection algorithm if there is no complete object frame information, and perform feature extraction and comparison on the region of interest to determine the information of the product; The device for assisting blind people in shopping is a wearable device, which is a glove. The method of determining whether the image perceived by the camera module contains complete object frame information based on the camera module includes: Based on the camera module on the glove, it is determined in real time whether the image perceived by the camera module has complete object frame information; If there is complete object frame information, optical character recognition is performed on the product image within each object frame of the image perceived by the camera module to determine the complete information of each product in the image perceived by the camera module, and the information of all products in the image perceived by the camera module is played through the voice device.

13. A shopping system for the blind, characterized in that: The assisted blind shopping system includes a processor; the processor is configured to execute instructions to implement the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that The computer storage medium stores a program file capable of implementing any one of the methods of claims 1-11.

Images