Text label image positioning method and device based on intelligent robot service platform

Abstract

The application provides a text label image positioning method and device based on an intelligent robot service platform, which is applied to a lost object searching robot. The method comprises the following steps: obtaining a lost object searching instruction, and determining a target object according to the lost object searching instruction; roughly positioning a target position according to a radio frequency signal from the target object; moving to the target position where the target object can be seen according to a navigation path from a current position to the target position; after reaching the target position, updating a common sense map in real time according to the lost object searching instruction and an environment image obtained by shooting at a preset time step; after each update, selecting a moving instruction by using the updated common sense map, and moving according to the moving instruction until the target object is reached. According to the scheme, in a complex environment where the target object is temporarily invisible, the lost object searching robot can also move to the target position where the target object can be seen according to the radio frequency signal from the target object. Therefore, the scheme can be used for searching for lost objects in a complex environment, and has a wider application range.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot searching, and in particular to a text label image positioning method and device based on an intelligent robot service platform. BACKGROUND

[0002] With the development of robots and artificial intelligence technology, robots equipped with artificial intelligence technology based on deep learning and neural networks are currently widely used in many daily life scenarios.

[0003] Among them, the robot searching scenario is a typical application scenario of robots. In this application scenario, the robot obtains a searching instruction for a target object, and then searches for the target object in the environment (such as a large warehouse, an office, or a residence) and moves to the location of the target object.

[0004] The existing implementation scheme for robot searching is to use a camera to capture environment images in real time, use an image recognition model to recognize and locate the target object in the environment images, and then determine a navigation path according to the location of the located target object and the current location of the robot, and finally move to the location of the target object according to the navigation path.

[0005] The problem with this implementation scheme is that in some complex environments, there may be many obstructions between the target object and the robot, which makes it impossible for the robot to recognize the target object from the captured environment images (the target object may not be displayed at all, or only a corner of the target object is displayed and cannot be recognized). In this case, the existing implementation scheme obviously cannot complete the searching. That is, the application range of the existing robot searching technology is limited, and it cannot implement searching in a complex environment with a large number of obstructions. SUMMARY

[0006] To overcome the shortcomings of the prior art, the present application provides a text label image positioning method and device based on an intelligent robot service platform to provide a robot searching scheme that can be applied to complex environments.

[0007] The first aspect of the present application provides a text label image positioning method based on an intelligent robot service platform, applied to a searching robot, the method comprising:

[0008] obtaining a searching instruction and determining a target object according to the searching instruction;

[0009] coarsely positioning according to the radio frequency signal emitted by the target radio frequency tag to obtain a target position; wherein the target radio frequency tag refers to a Radio Frequency Identification (RFID) tag fixed on the target object;

[0010] generate a navigation path from the current position of the object-searching robot to the target position, and move to the target position according to the navigation path; wherein, when the object-searching robot is at the target position, the target object is visible to the object-searching robot;

[0011] after the object-searching robot moves to the target position, update the common-sense map in real time according to the object-searching instruction and the environment image obtained by shooting at a preset time step;

[0012] after each update, select a real-time moving instruction by using the updated common-sense map, and move according to the real-time moving instruction until the target object is reached.

[0013] The second aspect of the present application provides a text label image positioning device based on an intelligent robot service platform, which is applied to an object-searching robot, and the device comprises:

[0014] an obtaining unit, configured to obtain an object-searching instruction, and determine a target object according to the object-searching instruction;

[0015] a positioning unit, configured to perform coarse positioning according to a radio frequency signal emitted by a target radio frequency tag, and obtain a target position; wherein, the target radio frequency tag refers to a radio frequency identification (RFID) tag fixed on the target object;

[0016] a navigation unit, configured to generate a navigation path from the current position of the object-searching robot to the target position, and move to the target position according to the navigation path; wherein, when the object-searching robot is at the target position, the target object is visible to the object-searching robot;

[0017] an updating unit, configured to, after the object-searching robot moves to the target position, update a common-sense map in real time according to the object-searching instruction and an environment image obtained by shooting at a preset time step;

[0018] a moving unit, configured to, after each update, select a moving instruction by using the updated common-sense map, and move according to the moving instruction until the target object is reached.

[0019] The application provides a text label image positioning method and device based on an intelligent robot service platform, which is applied to a lost object searching robot. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of the provided drawings.

[0021] Figure 1 A principle schematic diagram of a text label image positioning method based on an intelligent robot service platform provided by the embodiments of the present application;

[0022] Figure 2 A flowchart of a text label image positioning method based on an intelligent robot service platform provided by the embodiments of the present application;

[0023] Figure 3 A process schematic diagram of obtaining a first semantic feature provided by the embodiments of the present application;

[0024] Figure 4 A process schematic diagram of obtaining a second semantic feature provided by the embodiments of the present application;

[0025] Figure 5 A process schematic diagram of updating a common sense map provided by the embodiments of the present application;

[0026] Figure 6 A process schematic diagram of selecting a moving instruction provided by the embodiments of the present application;

[0027] Figure 7 A structure schematic diagram of a text label image positioning device based on an intelligent robot service platform provided by the embodiments of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0029] Please refer to Figure 1 A principle diagram of a text label image positioning method based on an intelligent robot service platform is provided for the embodiments of the present application. After the lost object robot is started, radio frequency signals, image signals and voice signals can be obtained from the outside. The radio frequency signals are emitted by radio frequency identification (RFID) tags pre-installed on various objects in the environment where the lost object robot is located. The image signals refer to the environment images captured by the lost object robot. The image signals can be captured by a monocular device or other visual acquisition devices. The voice signals are voice form lost object instructions input by a user, which contain the name of the target object to be found, such as "mobile phone" and "tea pot".

[0030] After the above signals are obtained, the lost object robot performs instruction coding and feature extraction on the received radio frequency signals, and then performs rough positioning on the target object according to the extracted features of the radio frequency signals to determine the target position.

[0031] After the target position is determined, the lost object robot generates a navigation path from the current position to the target position by using a path generation algorithm, and then navigates to the target position according to the navigation path. The path generation algorithm can be any existing robot path finding algorithm. The specific implementation process of the algorithm can be referred to the related prior art, and will not be described here. As an example, the lost object robot can generate a navigation path from the current position to the target position by using A-Star (also known as A*) algorithm. The A-Star algorithm is a direct search method for solving the shortest path in a static road network.

[0032] After moving to the target position, the lost object robot can gradually search and generate a path according to the environment images captured in real time by the camera, the pre-trained deep model (i.e. deep learning model) and the external knowledge base, and finally move to the end position, i.e. the position of the target object.

[0033] The radio frequency tags are fixed on certain objects. After the robot receives the instruction for finding the tagged objects, the rough positioning is obtained according to the keywords in the voice combined with the RFID signals.

[0034] The robot is navigated to the rough positioning point by using the A* algorithm to generate a path.

[0035] After reaching the rough positioning point, the object is further positioned using a specified framework combined with an external knowledge base to generate a path.

[0036] The main network of the specified framework is composed of four modules, scene frame parsing, text expression parsing, common sense graph learning, and action policy generation,

[0037] The scene frame parsing module and the text expression parsing module are responsible for parsing the image data and text data input into the network,

[0038] The information extracted after parsing is input into the common sense graph learning module, which is guided to continuously update, display some common sense structures, and highlight the relationships and weights between various objects. After obtaining the feature embedding of the common sense graph, it is connected with the attributes of the scene frame image and the text expression,

[0039] An action is extracted from the possible action set through the action policy module.

[0040] The method of the embodiment of the application will be described in detail below with reference to the accompanying drawings.

[0041] Please refer to Figure 2 A flowchart of a text label image positioning method based on an intelligent robot service platform provided by the embodiment of the application. The method can be applied to a lost-and-found robot, that is, each step in the method can be executed by the lost-and-found robot. The method can include the following steps.

[0042] First of all, it needs to be pointed out that before applying the method provided by the embodiment, an RFID tag for uniquely identifying each object needs to be set on each object in the application environment of the lost-and-found robot, so that the lost-and-found robot can determine from which object the radio frequency signal comes from by analyzing the radio frequency signal after receiving the radio frequency signal emitted by the RFID tag.

[0043] For example, when the lost-and-found robot is applied in a residential environment, different RFID tags can be pasted on the surfaces of objects appearing in the residence.

[0044] S201, obtaining a lost-and-found instruction, and determining a target object according to the lost-and-found instruction.

[0045] Optionally, the lost-and-found instruction is a voice lost-and-found instruction.

[0046] Correspondingly, the specific execution process of step S201 can include:

[0047] The voice recognition technology is used to recognize the instruction text corresponding to the voice lost-and-found instruction;

[0048] According to the keywords contained in the instruction text, the target object is determined.

[0049] The voice recognition technology can be any existing voice recognition technology, and the embodiment is not limited in the specific recognition process.

[0050] After obtaining the instruction text corresponding to the voice search instruction, at least one keyword contained in the instruction text can be recognized according to the pre-constructed vocabulary library. For example, when the environment where the search robot is located is a common house, the obtained instruction text can be "find the mobile phone next to the sofa", and according to the pre-constructed vocabulary library, the keywords "sofa", "next to", "find", and "mobile phone" can be recognized in the instruction text.

[0051] Finally, according to the pre-set instruction recognition rule and the order of each keyword in the instruction text, the keyword representing the target object is determined, so as to determine the target object to be searched. In combination with the above example, it is assumed that the instruction recognition rule is that the first keyword representing the object appearing after the keywords such as "find" and "locate" is the keyword representing the target object; according to the instruction recognition rule, the first keyword representing the object appearing after "find" in the above instruction text is "mobile phone", so the target object can be determined as the mobile phone to be searched this time.

[0052] S202, coarse positioning is performed according to the radio frequency signal emitted by the target radio frequency tag to obtain a target position.

[0053] The target radio frequency tag refers to a radio frequency identification (RFID) tag fixed on the target object.

[0054] The search robot is pre-installed with an RFID reading device for receiving radio frequency signals. The device can be installed on the left and right sides of the search robot, that is, at least two RFID reading devices are installed on the search robot. These RFID reading devices can receive radio frequency signals emitted by RFID tags in the environment in real time during the movement of the search robot, achieving the effect of simulating multiple RFID device positioning.

[0055] The execution process of step S202 can include:

[0056] A1, determining the first signal strength and the first azimuth angle of the radio frequency signal emitted by the target radio frequency tag, and determining the distance and the azimuth of the target object relative to the search robot according to the first signal strength and the first azimuth angle;

[0057] A2, determining a second signal strength and a second azimuth angle of the radio frequency signal emitted by the other radio frequency tags, and determining the distance and the azimuth of the other object relative to the object-searching robot according to the second signal strength and the second azimuth angle; wherein the other radio frequency tags refer to the radio frequency tags fixed on the other objects; and the other objects refer to the objects other than the target object;

[0058] A3, determining the target position according to the distance and the azimuth of the target object relative to the object-searching robot, the distance and the azimuth of the other object relative to the object-searching robot, and the pre-input environment map data.

[0059] It should be noted that steps A1 and A2 can be executed simultaneously, or A2 can be executed first and then A1, and the execution order of the two steps is not limited.

[0060] In A1 and A2, the object-searching robot can analyze the received radio frequency signal to obtain the instruction code carried by the radio frequency signal, and then determine whether the received radio frequency signal is from the target radio frequency tag or the other radio frequency tag according to the preset instruction code and the corresponding relationship between the objects.

[0061] After determining the source of the radio frequency signal, the object-searching robot can regard the radio frequency signals received by the two RFID reading devices at different times of the moving process as the radio frequency signals received by three or more RFID reading devices at the same time, and then extract the characteristics of the radio frequency signals according to the existing three-point positioning technology to obtain the signal strength and the azimuth angle corresponding to the radio frequency signals, and further determine the distance and the azimuth of the object at the signal source relative to the object-searching robot according to the signal strength and the azimuth angle.

[0062] The three-point positioning technology is a mature existing technical means for determining the position of a signal source according to signals received at different positions, and its specific implementation process is not described again.

[0063] The above process is described below with the example of the radio frequency signal of the target radio frequency tag. The lost-and-found robot determines that the radio frequency signal comes from the target radio frequency tag according to the instruction code carried by the radio frequency signal. Then, the lost-and-found robot determines the radio frequency signals of at least four target radio frequency tags received by the two RFID reading devices in at least two adjacent time points as the radio frequency signals of the target radio frequency tags received by the four different position RFID reading devices. Then, the lost-and-found robot determines the signal strength of each of the four radio frequency signals (i.e., the first signal strength), determines the azimuth of the four radio frequency signals (i.e., the first azimuth) according to the signal strength of the four radio frequency signals and the position of the RFID reading device where the four radio frequency signals are received by using the three-point positioning technology. Finally, the distance and the azimuth of the target object where the target radio frequency tag is arranged relative to the lost-and-found robot are determined according to the first signal strength and the first azimuth of the four radio frequency signals from the target radio frequency tag. The distance can be determined according to the signal strength of the radio frequency signal.

[0064] The position of the RFID reading device where the four radio frequency signals are received can be determined according to the historical moving track recorded by the lost-and-found robot in real time and the installation position of the RFID reading device on the lost-and-found robot.

[0065] Similarly, after the lost-and-found robot determines that the radio frequency signal comes from other radio frequency tags according to the instruction code carried by the radio frequency signal, the signal strength (i.e., the second signal strength) and the azimuth (i.e., the second azimuth) of the radio frequency signal emitted by the other radio frequency tags can be determined according to the above method, and then the distance and the azimuth of the other object relative to the lost-and-found robot are determined according to the second signal strength and the second azimuth.

[0066] Steps A1 and A2 are equivalent to Figure 1 The instruction code and the feature extraction shown in the figure, wherein the signal strength and the azimuth of the radio frequency signal are equivalent to the signal features of the radio frequency signal obtained by the feature extraction.

[0067] A3, according to the distance and the azimuth of the target object relative to the lost-and-found robot, the distance and the azimuth of the other object relative to the lost-and-found robot, and the pre-input environment map data, the target position is determined.

[0068] The pre-input environment map data can include the size data of each object in the environment and the data of the movable path in the environment.

[0069] In step A3, after obtaining the distance and orientation of the target object and other objects in the environment relative to the search robot, the search robot can determine the positions of the target object and other objects in the coordinate system with the search robot as the origin, and then analyze the relative position relationship and occlusion relationship between the target object and other objects according to the size data of the target object and other objects in the environment map data, for example, analyze that the target object is located on the left side of other object A and is partially occluded by other object A, is located on the right side of other object B and is completely occluded by other object B, etc.

[0070] Then, the search robot determines at least one candidate target position according to the above relative position relationship and occlusion relationship, and these candidate target positions all satisfy the condition that the target object is visible to the search robot when the search robot reaches the candidate target position, in other words, the search robot can directly see the target object at the candidate target position.

[0071] Wherein, the target object being visible to the search robot can be defined as at least 80% of the surface of the target object displayed in the environment image captured by the search robot. Here, 80% is a pre-set proportion threshold, which can be adjusted according to needs in actual application, for example, if higher positioning accuracy is required, it can be adjusted to 90%.

[0072] After determining the candidate target positions, the search robot can determine which of these candidate target positions can be reached along the movable path according to the data of the movable path, if only one candidate target position satisfies the condition of being able to be reached along the movable path, the candidate target position is determined as the target position, and if there are multiple candidate target positions satisfying the condition of being able to be reached along the movable path, the candidate target position closest to the search robot is determined as the target position.

[0073] S203, generating a navigation path from the current position of the search robot to the target position, and moving to the target position according to the navigation path.

[0074] Wherein, when the search robot is at the target position, the target object is visible to the search robot.

[0075] As described above, after obtaining the target position, the search robot can use any existing path generation algorithm including but not limited to A-Star algorithm to generate a navigation path from the current position to the target position, and then move to the target position according to the instructions of the navigation path.

[0076] Therefore, the specific process of generating the navigation path can refer to the related existing algorithm, which will not be described here.

[0077] S204, after the object-searching robot moves to the target position, updating the common-sense map in real time according to the object-searching instruction and the environment image obtained by shooting at a preset time step.

[0078] S205, after each update, selecting a real-time moving instruction by using the updated common-sense map, and moving according to the real-time moving instruction until the target object is reached.

[0079] Steps S204 and S205 can be executed once every preset time step after the object-searching robot moves to the target position and before the object-searching robot moves to the position of the target object.

[0080] That is, after reaching the target position, the object-searching robot shoots an environment image every preset time step, updates the common-sense map according to the environment image shot this time, then executes S205 according to the updated common-sense map this time to determine the real-time moving instruction this time, and finally moves according to the real-time moving instruction this time.

[0081] For the convenience of description, the execution process of steps S204 and S205 will be described below by taking the tth execution of steps S204 and S205 during the movement of the object-searching robot from the target position to the terminal position as an example.

[0082] First, steps S204 and S205 can be executed by a specified framework pre-established on the object-searching robot, which can include a text expression analysis module, a scene frame image analysis module, a common-sense map learning module, and an action strategy generation module.

[0083] The specific execution process of step S204 can include:

[0084] B1, converting the instruction text into an instruction text expression; wherein the instruction text expression includes a subject, a relationship, and an object;

[0085] B2, analyzing the instruction text expression by using a scene image analysis tool to obtain a first semantic feature corresponding to the instruction text expression;

[0086] B3, shooting an environment image in real time at a preset time step;

[0087] B4, after obtaining the environment image each time, determining a second semantic attribute of the environment image according to the environment image and a pre-constructed object image database, and analyzing the second semantic attribute to obtain a second semantic feature;

[0088] B5, updating the common-sense map according to the first semantic feature and the second semantic feature; wherein, at the first update, the common-sense map is a preset initial common-sense map.

[0089] Steps B1 and B2 can be executed by the aforementioned text expression analysis module. Please refer toFigure 3 For the process of obtaining the first semantic feature, it can be seen that the process of obtaining the first semantic feature described in steps B1 and B2 can be subdivided into four links of instruction text input, content initialization parsing, semantic attribute extraction and first semantic feature obtaining, wherein the instruction text input and content initialization parsing correspond to step B1, and the semantic attribute extraction and first semantic feature obtaining correspond to step B2.

[0090] In B1, the text expression parsing module can obtain the instruction text recognized from the voice in S201 as the input text data, and then the text expression parsing module performs content initialization parsing on the instruction text to obtain a regular type text expression L. The form of the text expression can be "subject, relationship, object", which is equivalent to parsing the text instruction into a text expression composed of subject words, relationship words and object words, or "subject, none, none", which is equivalent to parsing the instruction text into a text expression containing only subject words.

[0091] The specific parsing process can be referred to the related prior art, which will not be described here.

[0092] In B2, the text expression parsing module can process the text expression L obtained in B1 based on the existing streaming graph partition (SGP) algorithm to obtain the first semantic attribute A corresponding to the text expression. L The first semantic attribute can be composed of the feature vector of each word in the text expression, and the dimension of the feature vector of each word is 100.

[0093] Optionally, the streaming graph partition algorithm in the embodiment can be based on the existing word vector model, for example, based on the Glove model, and correspondingly, the feature vector of each word in the first semantic attribute obtained by the streaming graph partition algorithm is equivalent to the 100-dimensional embedding vector output by the Glove model.

[0094] For example, when the text expression includes a subject, a relationship and an object, the process of extracting the first semantic attribute from the text expression can be represented by the following formula:

[0095] A L =SGP(L)=(L SUB ,L RELA ,L OB )

[0096] In the formula, SGP represents the streaming graph partition algorithm, L SUB represents the feature vector of the subject word, L RELA represents the feature vector of the relationship word, and L OB represents the feature vector of the object word.

[0097] After obtaining the first semantic attribute, the first semantic attribute can be mapped into a 512-dimensional feature vector by using a fully connected (FC) layer and a rectified linear unit (ReLU) pre-set in the text expression parsing module. The 512-dimensional vector after mapping is the first semantic feature obtained by B2, denoted as E AL .

[0098] The first semantic feature E AL participates in the construction and update of the common sense graph together with the second semantic feature in subsequent work, and finally participates in the generation of action strategy in the form of embedding vector.

[0099] Step B4 can be performed by the aforementioned scene frame image parsing module.

[0100] In step B3, for the convenience of description, the environment image taken by the lost object searching robot for the t-th time since arriving at the target position can be denoted as I t . For example, the preset time step can be 0.5 seconds. After arriving at the target position, the lost object searching robot takes the environment image I1 for the first time after 0.5 seconds, takes the environment image I2 after another 0.5 seconds, and so on.

[0101] For the convenience of description, the execution process of step B4 will be described below by taking the environment image I t taken for the t-th time as an example.

[0102] Please refer to Figure 4 , the process of obtaining the second semantic feature in step B4 can include three links of environment image input, scene frame image parsing, and obtaining the second semantic feature of the environment image.

[0103] The environment image input is to read the environment image taken in step B3 from the memory of the lost object searching robot.

[0104] The scene frame image parsing is to parse the obtained environment image I t according to the pre-constructed object image database to obtain the second semantic attribute A It corresponding to the environment image.

[0105] The object image database is equivalent to the external knowledge base shown in Figure 1 . The object image database pre-stores images of various objects in the environment of the lost object searching robot.

[0106] For example, when the environment in which the object-finding robot is located is a residential house, the aforementioned object image database may include images of various furniture and furniture items that frequently appear in residential houses, such as tables, chairs, coffee tables, slippers, and sofas.

[0107] The scene frame image parsing module can first use the aforementioned object image database to train an existing object detection network, such as YOLOv3. After training, the scene frame image parsing module can then call the trained object detection network to process the environment image I. t Obtain the second semantic attribute A corresponding to the environmental image. It .

[0108] Environmental Image I t The second semantic attribute can include the environmental image I. t The location information (including the coordinates and dimensions of the bounding box) and labels of multiple bounding boxes used to indicate the location of various objects are used to represent them.

[0109] The coordinates of the bounding box can be represented by the coordinates (x, y) of the center point of the bounding box. The size can include the height (h) and width (w) of the bounding box. Thus, the position information of a bounding box can be recorded as [x, y, w, h].

[0110] Furthermore, if the environmental image I t The objects displayed are numbered i, where i ranges from 1 to N, and N is the environment image I. t The total number of objects displayed in the environment image I t The second semantic attribute A It This can be represented as:

[0111] (label i [x,y,w,h] i |i∈I t ).

[0112] Where, label i Represents the environment image I t The label of the i-th object in the array, for example, if the i-th object is a table, then the label is... i It means "table", [x, y, w, h] i Indicates the first i The position information of the bounding box of an object.

[0113] The above-mentioned acquisition of environmental image I t The corresponding second semantic attribute A It The process can be represented by the following formula:

[0114] AI t=YOLO(I t ) = (label i [x,y,w,h] i |i∈I t ).

[0115] In the formula, YOLO represents the object detection network called by the scene frame image parsing module.

[0116] When calling the object detection network to parse and obtain the second semantic attribute, the scene frame image parsing module can first construct a 300*600 zero matrix, and then use the environment image I identified by the object detection network. t The position information and labels of each bounding object are written into the zero matrix to obtain the corresponding second semantic attribute.

[0117] In obtaining environmental image I t After determining the second semantic attribute, the scene frame image parsing module can first map the second semantic attribute to a 2048-dimensional feature vector, and then use the environment image I... t The first two frames of the environment image, that is, I t-1 and I t-2 The 2048-dimensional feature vectors mapped from the second semantic attribute are concatenated together to obtain the concatenated vector, which is the environment image I. t The second semantic feature E It .

[0118] The above process can be expressed by the following formula:

[0119] E It =Concat(A t A t-1 A t-2 ).

[0120] In the above formula, A t Represents the environment image I t The 2048-dimensional feature vector obtained by mapping the second semantic attribute, A t-1 Represents the environment image I t-1 The 2048-dimensional feature vector obtained by mapping the second semantic attribute, A t-2 Represents the environment image I t-2 The second semantic attribute mapping yields a 2048-dimensional feature vector. `contact` represents concatenating these three feature vectors end-to-end into a single vector.

[0121] Step B5 can be performed by the aforementioned common sense graph learning module.

[0122] The execution process of step B5 can be found in [reference needed]. Figure 5It can be seen that step B5 includes common sense graph definition, initialization node, calculation of the position relationship between objects in the environment image through the boundary object box, and common sense graph updating.

[0123] In the common sense graph updating link, the specific updating method is that if the position relationship does not exist in the common sense graph, the corresponding edge is added, and the weight is initialized to 1; if the position relationship exists in the common sense graph, the weight of the corresponding edge is added by 1.

[0124] It should be noted that the two links of common sense graph definition and initialization node belong to the process of common sense graph initialization, that is, only when step B5 is executed for the first time, common sense graph definition and initialization node are needed, and after step B5 is executed, it is only necessary to continue updating on the basis of the common sense graph updated last time.

[0125] In the common sense graph definition link, the common sense graph learning module defines the common sense graph as G=(N, E), where N represents the set of all nodes in the common sense graph, each node in the set represents an object in the environment of the object searching robot, and E represents the set of edges in the common sense graph, which is used to represent the position relationship between the objects corresponding to each two nodes. Each edge has a corresponding weight, and the weight of the edge is determined according to the position relationship between the objects corresponding to the two nodes connected by the edge.

[0126] Optionally, the above common sense graph can be displayed on a visual interface, and when displayed on the visual interface, the length of each edge can be determined according to the weight of the edge, the greater the weight of the edge, the shorter the length, and the closer the two nodes connected by the corresponding edge, thereby reflecting the position relationship between the objects corresponding to the two nodes. Specifically, when visualizing, the length of an edge can be determined according to the reciprocal of the weight W of the edge, that is, 1 / W, for example, multiplying 1 / W by a predetermined constant to determine the length of the edge as the product.

[0127] In the initialization node step, the common sense graph learning module creates nodes corresponding to objects in the common sense graph according to the first semantic attribute and the second semantic attribute obtained in the foregoing steps. For example, the feature vectors of the subject words and the object words in the first semantic attribute both point to a specific object in the environment, and each label in the second semantic attribute also represents a specific object in the environment. Therefore, in the node definition step, the common sense graph learning module identifies the feature vectors of the subject words and the feature vectors of the object words in the first semantic attribute one by one, and creates a node corresponding to an object represented by each feature vector in the common sense graph as the feature vector is identified. The common sense graph learning module also identifies the labels in the second semantic attribute one by one, and creates a node corresponding to an object represented by each label in the common sense graph as the label is identified. If a duplicate node is found during the creation process, the duplicate node is merged into one node, so as to finally ensure that the nodes in the common sense graph correspond one-to-one to the objects in the environment of the object-searching robot.

[0128] In the step of calculating the positional relationship between objects in the environment image through the boundary object frame, the common sense graph learning module can compare the positional information and the size of the boundary object frames of each two objects in the environment image I t shot this time, and determine the positional relationship matching the comparison result from the preset multiple positional relationships according to the comparison result.

[0129] Specifically, in the common sense graph learning module, the encodings "0000" to "1111" can be preset in sequence, and the 16 encodings represent the positional relationship of two objects in four directions of the boundary object frames of the two objects in sequence, that is, 16 kinds of positional relationships are preset in total, and each kind of positional relationship corresponds to one comparison result between two boundary object frames.

[0130] For example, for the two objects A and B shown in the environment image I t , the positional information and the size of the boundary object frame of A are denoted as Sa(x a , y a , w a , h a ), and the positional information and the size of the boundary object frame of B are denoted as Sb(x b , y b , w b , h b ). If the comparison result of Sa and Sb is x a < x b , y a < y b , w a < w b , and h a < h bIf the position relationship between object A and object B is determined as “0000”, other position relationships can be inferred according to the example, and details are not described herein.

[0131] Therefore, each time step B5 is executed, the common sense graph learning module can determine the position relationship between object A and object B according to the environment image I t The comparison result between the displayed object boundary object frames determines the position relationship between object A and object B in the environment image I t The position relationship between each pair of displayed objects is determined.

[0132] Finally, the common sense graph learning module can update the weight of the edge in the common sense graph according to the position relationship determined above, such as Figure 5 As shown in the figure, if the determined position relationship between the two objects does not have a corresponding edge in the common sense graph (i.e., the edge does not exist), a new edge representing the position relationship is added between the nodes corresponding to the two objects in the common sense graph, and the weight of the new edge is set to 1; if the determined position relationship between the two objects has a corresponding edge in the common sense graph (i.e., the edge exists), the weight of the corresponding edge is increased by 1 based on the original weight.

[0133] The execution process of step S205 can include the following steps:

[0134] C1, determining a node matrix according to the node feature vectors of the nodes in the updated common sense graph, and determining an edge matrix according to the weights of the edges in the updated common sense graph;

[0135] C2, calculating a common sense graph feature of the updated common sense graph based on the node matrix and the edge matrix;

[0136] C3, selecting a candidate moving instruction as a real-time moving instruction according to the common sense graph feature.

[0137] Step S205 can be specifically executed by an action policy generation module.

[0138] The execution process of step S205 can be seen from Figure 6 It can be seen that when the action policy generation module executes S205, it first obtains a common sense graph feature according to the updated common sense graph, splices the common sense graph feature, the first semantic feature, and the second semantic feature to form a feature vector, then processes the feature vector using a pre-trained deep model to obtain a corresponding moving strategy set, and finally selects a real-time moving instruction based on the moving strategy set.

[0139] When obtaining the common sense graph feature, that is, when executing C1 and C2, the action policy generation module first collects the node feature vectors of all nodes in the updated common sense graph into a matrix to form the node matrix in C1, and the node matrix is denoted as X=[x1, x2……x n], x i (i is an integer from 1 to n) represents the node feature vector of the i-th node in the updated common sense graph, and n represents the total number of nodes in the common sense graph.

[0140] The node feature vector of each node in the common sense graph can be calculated by using a graph embedding algorithm on the updated common sense graph. The graph embedding algorithm is a mature existing algorithm, and its specific implementation will not be described again.

[0141] When determining the edge matrix, the action policy generation module can first construct a unit matrix with a size of n*n*16, and then write the inverse numbers of the weights of all edges in the updated common sense graph into the matrix, thereby obtaining the edge matrix A in C1.

[0142] When performing C2, the action policy generation module calculates the common sense graph feature E of the updated common sense graph by using the following formula G :

[0143] E G =A*X T .

[0144] When performing C3, the action policy generation module can directly perform feature attribute splicing on the common sense graph feature and the aforementioned first semantic feature, second semantic feature after obtaining the common sense graph feature.

[0145] In some optional embodiments, in order to reduce the amount of calculation, after obtaining the common sense graph feature, the action policy generation module can first map the common sense graph feature into a 512-dimensional vector, and then perform feature attribute splicing on the mapped 512-dimensional common sense graph feature and the first semantic feature E AL and the second semantic feature E It to obtain a 1536-dimensional semantic common sense feature vector E.

[0146] After obtaining the semantic common sense feature vector, the action policy generation module can calculate the semantic common sense feature vector based on the Asynchronous Advantage Actor-Critic (A3C) algorithm to obtain a set of movement strategies. The above process can be represented by the following formula:

[0147] p t ~Π(E; θ) = Π(concat(E AL ,E It ,E G ); θ).

[0148] In the above formula, Π() represents the A3C algorithm, and θ represents the set of candidate movement commands, which includes multiple candidate movement commands determined based on the current position of the finding robot and its surrounding environment. For example, if the finding robot can currently move in four directions—forward, backward, left, and right—then the set of candidate movement commands could include four options: "move forward," "move backward," "move left," and "move right." AL E It E G ) represents the semantic commonsense feature vector obtained by concatenating the commonsense graph features, the first semantic feature, and the second semantic feature. p t This represents the real-time movement command selected based on the environmental image obtained from the t-th capture.

[0149] Based on the execution process of S204 and S205 above, it can be understood that after the locator reaches the target location, the locator can select a real-time movement command based on the environmental image around the locator at every time step, and move according to the real-time movement command until a new real-time movement command is selected after one time step. This process continues until the locator finally moves to the destination location, which is the location of the target object.

[0150] This application provides a text-label image localization method based on an intelligent robot service platform, applied to a finding robot. The method includes: obtaining a finding instruction and determining the target object based on the instruction; roughly locating the target position based on the radio frequency signal from the target object; moving to the target position where the target object is visible according to the navigation path from the current position to the target position; after reaching the target position, updating the common sense map in real time according to the finding instruction and captured environmental images at preset time steps; after each update, selecting a movement instruction using the updated common sense map and moving according to the movement instruction until the target object is located. With this solution, even in complex environments where the target object is temporarily invisible, the finding robot can move to the target position where the target object is visible based on the radio frequency signal from the target object. Therefore, this solution can be used for finding objects in complex environments and has a wider range of applications.

[0151] This solution uses physical signal localization and dynamic motion generation algorithms to enable robots to automatically search for target objects and generate paths in complex environments, utilizing a specified framework and a learnable external knowledge base. Even if objects are not visible in the scene, the robot can still complete the predetermined task.

[0152] According to the text label image positioning method based on the intelligent robot service platform provided in the embodiment of the application, the embodiment of the application further provides a text label image positioning device based on the intelligent robot service platform, which is applied to a lost-and-found robot, that is, the device can be understood as a control device carried by the lost-and-found robot. Please refer to Figure 7 The device can include the following units.

[0153] The obtaining unit 701 is configured to obtain a lost-and-found instruction and determine a target object according to the lost-and-found instruction.

[0154] The positioning unit 702 is configured to perform coarse positioning according to a radio frequency signal emitted by a target radio frequency tag to obtain a target position. The target radio frequency tag refers to a radio frequency identification (RFID) tag fixed on the target object.

[0155] The navigation unit 703 is configured to generate a navigation path from a current position of the lost-and-found robot to the target position, and move to the target position according to the navigation path. When the lost-and-found robot is located at the target position, the target object is visible to the lost-and-found robot.

[0156] The updating unit 704 is configured to update the common sense map according to the lost-and-found instruction and the obtained environment image in real time at a preset time step after the lost-and-found robot moves to the target position.

[0157] The moving unit 705 is configured to select a moving instruction by using the updated common sense map after each update, and move according to the moving instruction until the target object is located.

[0158] Optionally, the lost-and-found instruction is a voice lost-and-found instruction.

[0159] When the obtaining unit 701 determines the target object according to the lost-and-found instruction, the obtaining unit 701 is specifically configured to:

[0160] An instruction text corresponding to the voice lost-and-found instruction is obtained by using a voice recognition technology.

[0161] The target object is determined according to a keyword contained in the instruction text.

[0162] Optionally, when the positioning unit 702 performs coarse positioning according to the radio frequency signal emitted by the target radio frequency tag to obtain the target position, the positioning unit 702 is specifically configured to:

[0163] A first signal strength and a first azimuth angle of the radio frequency signal emitted by the target radio frequency tag are determined, and the distance and the azimuth of the target object relative to the lost-and-found robot are determined according to the first signal strength and the first azimuth angle.

[0164] determine a second signal strength and a second azimuth angle of a radio frequency signal emitted by another radio frequency tag, and determine a distance and an azimuth of the other object relative to the object-searching robot according to the second signal strength and the second azimuth angle; wherein the other radio frequency tag refers to a radio frequency tag fixed on the other object; and the other object refers to an object other than the target object;

[0165] According to the distance and the azimuth of the target object relative to the object-searching robot, the distance and the azimuth of the other object relative to the object-searching robot, and the pre-input environment map data, the target position is determined.

[0166] Optionally, the updating unit 704 updates the common sense graph in real time according to the object-searching instruction and the environment image obtained by shooting at a preset time step, and specifically for:

[0167] converts the instruction text into an instruction text expression; wherein the instruction text expression includes a subject, a relationship and an object;

[0168] analyzes the instruction text expression by using a scene graph analysis tool to obtain a first semantic feature corresponding to the instruction text expression;

[0169] shoots the environment image in real time at a preset time step;

[0170] after obtaining the environment image each time, determines a second semantic attribute of the environment image according to the environment image and a pre-constructed object image database, and analyzes the second semantic attribute to obtain a second semantic feature;

[0171] updates the common sense graph according to the first semantic feature and the second semantic feature; wherein when updating for the first time, the common sense graph is a preset initial common sense graph.

[0172] Optionally, when the moving unit 705 selects the moving instruction by using the updated common sense graph, specifically for:

[0173] determines a node matrix according to a node feature vector of a node in the updated common sense graph, and determines an edge matrix according to a weight of an edge in the updated common sense graph;

[0174] calculates a common sense graph feature of the updated common sense graph based on the node matrix and the edge matrix;

[0175] selects one of the candidate moving instructions as the real-time moving instruction according to the common sense graph feature.

[0176] The specific principle and beneficial effects of the text label image positioning device based on the intelligent robot service platform provided in this embodiment can be referred to the related steps and beneficial effects of the text label image positioning method based on the intelligent robot service platform provided in this embodiment, and will not be repeated here.

[0177] Finally, it needs to be pointed out that in this document, the terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or sequence between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0178] It should be noted that the "first", "second", and the like concepts mentioned in the present application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0179] A person skilled in the art can implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text-label image localization method based on an intelligent robot service platform, characterized in that, Applied to a finding robot, the method includes: Receive a locator command and determine the target object based on the locator command; The first signal strength and first azimuth angle of the radio frequency signal emitted by the target RFID tag are determined, and the distance and orientation of the target object relative to the finding robot are determined based on the first signal strength and the first azimuth angle; wherein, the target RFID tag refers to a radio frequency identification (RFID) tag fixed on the target object; The second signal strength and second azimuth angle of the radio frequency signals emitted by other RFID tags are determined, and the distance and orientation of other objects relative to the finding robot are determined based on the second signal strength and the second azimuth angle; wherein, the other RFID tags refer to RFID tags fixed on other objects; the other objects refer to objects other than the target object; Based on the distance and orientation of the target object relative to the finding robot, the distance and orientation of other objects relative to the finding robot, and the size data of the target object and other objects in the pre-input environmental map data, the relative positional relationship and occlusion relationship between the target object and other objects are analyzed. Based on the relative positional relationship and occlusion relationship, at least one alternative target position is determined, and a target position is determined from at least one of the alternative target positions. The alternative target positions satisfy the condition that the target object is visible to the search robot when the search robot arrives at the alternative target position. Generate a navigation path from the current position of the object-finding robot to the target position, and move to the target position according to the navigation path; wherein, when the object-finding robot is at the target position, the target object is visible to the object-finding robot; After the locator moves to the target location, the common sense map is updated in real time according to the locator command and the captured environmental images at a preset time step. After each update, a real-time movement command is selected using the updated common sense graph, and the user moves according to the real-time movement command until the target object is located.

2. The method according to claim 1, characterized in that, The item locating command is a voice-based item locating command; Determining the target object according to the object-finding instruction includes: The command text corresponding to the voice-guided object search command is obtained by using speech recognition technology. The target object is determined based on the keywords contained in the instruction text.

3. The method according to claim 2, characterized in that, The step of updating the common sense map in real time according to the object-finding command and the captured environmental images at a preset time step includes: The instruction text is converted into an instruction text expression; wherein the instruction text expression includes a subject, a relation, and an object; The instruction text expression is parsed using a scene graph parsing tool to obtain the first semantic feature corresponding to the instruction text expression; Capture environmental images in real time according to a preset time step; After each environmental image is captured, the second semantic attribute of the environmental image is determined based on the environmental image and a pre-built object image database, and the second semantic attribute is parsed to obtain the second semantic feature; The commonsense graph is updated based on the first semantic feature and the second semantic feature; wherein, during the first update, the commonsense graph is a preset initial commonsense graph.

4. The method according to claim 1, characterized in that, The step of selecting a move command using the updated commonsense graph includes: The node matrix is ​​determined based on the node feature vectors of the nodes in the updated commonsense graph, and the edge matrix is ​​determined based on the edge weights in the updated commonsense graph. The updated common sense graph features are calculated based on the node matrix and the edge matrix. Based on the aforementioned common sense graph features, an alternative movement command is selected as the real-time movement command.

5. A text-label image positioning device based on an intelligent robot service platform, characterized in that, The device, used in a locator robot, includes: The acquisition unit is used to acquire a locator command and determine the target object based on the locator command. Positioning unit, used for: The first signal strength and first azimuth angle of the radio frequency signal emitted by the target RFID tag are determined, and the distance and orientation of the target object relative to the finding robot are determined based on the first signal strength and the first azimuth angle; wherein, the target RFID tag refers to a radio frequency identification (RFID) tag fixed on the target object; The second signal strength and second azimuth angle of the radio frequency signals emitted by other RFID tags are determined, and the distance and orientation of other objects relative to the finding robot are determined based on the second signal strength and the second azimuth angle; wherein, the other RFID tags refer to RFID tags fixed on other objects; the other objects refer to objects other than the target object; Based on the distance and orientation of the target object relative to the finding robot, the distance and orientation of other objects relative to the finding robot, and the size data of the target object and other objects in the pre-input environmental map data, the relative positional relationship and occlusion relationship between the target object and other objects are analyzed. Based on the relative positional relationship and occlusion relationship, at least one alternative target position is determined, and a target position is determined from at least one of the alternative target positions. The alternative target positions satisfy the condition that the target object is visible to the search robot when the search robot arrives at the alternative target position. A navigation unit is configured to generate a navigation path from the current position of the retrieval robot to the target position, and move to the target position according to the navigation path; wherein, when the retrieval robot is at the target position, the target object is visible to the retrieval robot; The update unit is used to update the common sense map in real time according to the retrieval instructions and the captured environmental images after the retrieval robot moves to the target location, according to a preset time step. The movement unit is used to select a movement command using the updated common sense graph after each update, and move according to the movement command until it reaches the location of the target object.

6. The apparatus according to claim 5, characterized in that, The item locating command is a voice-based item locating command; When the obtaining unit determines the target object according to the object-finding instruction, it is specifically used for: The command text corresponding to the voice-guided object search command is obtained by using speech recognition technology. The target object is determined based on the keywords contained in the instruction text.

7. The apparatus according to claim 6, characterized in that, When the updating unit updates the common sense map in real time according to the object-finding command and the captured environmental images at a preset time step, it is specifically used for: The instruction text is converted into an instruction text expression; wherein the instruction text expression includes a subject, a relation, and an object; The instruction text expression is parsed using a scene graph parsing tool to obtain the first semantic feature corresponding to the instruction text expression; Capture environmental images in real time according to a preset time step; After each environmental image is captured, the second semantic attribute of the environmental image is determined based on the environmental image and a pre-built object image database, and the second semantic attribute is parsed to obtain the second semantic feature; The commonsense graph is updated based on the first semantic feature and the second semantic feature; wherein, during the first update, the commonsense graph is a preset initial commonsense graph.

8. The apparatus according to claim 5, characterized in that, When the moving unit selects a moving command using the updated commonsense graph, it is specifically used for: The node matrix is ​​determined based on the node feature vectors of the nodes in the updated commonsense graph, and the edge matrix is ​​determined based on the edge weights in the updated commonsense graph. The updated common sense graph features are calculated based on the node matrix and the edge matrix. Based on the aforementioned common sense graph features, an alternative movement command is selected as the real-time movement command.

Images