A path planning method, device, electronic device and storage medium

Abstract

The embodiments of the present invention disclose a path planning method, device, electronic device and storage medium. The method may include: in response to a path planning instruction, obtaining path planning data, identifying the path planning data, and generating an initial path according to a first recognition result, wherein the path planning data includes part data and / or voice data for path planning, and the part data is obtained by collecting the target part; obtaining brain-computer data for adjusting the initial path, identifying the brain-computer data, and obtaining path adjustment information according to the second recognition result; adjusting the initial path based on the path adjustment information to obtain the target path. The technical solution of the embodiment of the present invention achieves the effect of flexible path planning, thereby meeting the demand for flexible control of mobile robots in reality.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of automatic control technology, and in particular to a path planning method, device, electronic device and storage medium. Background Art

[0002] With the development of science and technology, mobile robots have been widely used in home services, logistics warehousing, scene detection, etc. In order to enable mobile robots to efficiently complete various complex tasks, a path can be pre-planned according to the task, so that the mobile robot can move along the path to complete the task.

[0003] However, current path planning can only be achieved through pre-set paths, which cannot meet the needs of flexible control of mobile robots in reality. Summary of the invention

[0004] The embodiments of the present invention provide a path planning method, device, electronic device and storage medium to achieve the effect of flexible path planning.

[0005] According to one aspect of the present invention, a path planning method is provided, which may include:

[0006] In response to the path planning instruction, acquiring path planning data, identifying the path planning data, and generating an initial path according to a first identification result obtained, wherein the path planning data may include position data and / or voice data for path planning, and the position data is obtained by collecting the target position;

[0007] Acquiring brain-computer data for adjusting the initial path, identifying the brain-computer data, and obtaining path adjustment information according to a second identification result;

[0008] The initial path is adjusted based on the path adjustment information to obtain the target path.

[0009] According to another aspect of the present invention, a path planning device is provided, which may include:

[0010] an initial path generation module, configured to obtain path planning data in response to a path planning instruction, identify the path planning data, and generate an initial path according to a first identification result obtained, wherein the path planning data includes position data and / or voice data used for path planning, and the position data is obtained by collecting a target position;

[0011] A path adjustment information obtaining module, used to obtain brain-computer data for adjusting the initial path, identify the brain-computer data, and obtain path adjustment information according to a second identification result obtained;

[0012] The target path obtaining module is used to adjust the initial path based on the path adjustment information to obtain the target path.

[0013] According to another aspect of the present invention, there is provided an electronic device, which may include:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the path planning method provided by any embodiment of the present invention is implemented when the at least one processor executes the computer program.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer instructions are stored. The computer instructions are used to enable a processor to implement the path planning method provided by any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention obtains path planning data in response to a path planning instruction, identifies the path planning data, and generates an initial path according to the first identification result obtained, wherein the path planning data includes part data and / or voice data for path planning, and the part data is obtained by collecting the target part; obtains brain-computer data for adjusting the initial path, identifies the brain-computer data, and obtains path adjustment information according to the second identification result obtained; adjusts the initial path based on the path adjustment information to obtain the target path. The above technical solution does not need to perform path planning through a pre-set path, and can generate an initial path in real time according to the acquired path planning data, and then generate path adjustment information according to the acquired brain-computer data to adjust the initial path, thereby achieving the effect of flexible path planning, thereby meeting the needs of flexible control of mobile robots in reality.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1is a flow chart of a path planning method provided according to an embodiment of the present invention;

[0022] Figure 2 is a flow chart of another path planning method provided according to an embodiment of the present invention;

[0023] Figure 3 is a flowchart of an example of initial path generation in another path planning method provided by an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of an example of three-dimensional space modeling in another path planning method provided in an embodiment of the present invention;

[0025] Figure 5 is a flow chart of another path planning method provided according to an embodiment of the present invention;

[0026] Figure 6 is a flowchart of an example of brain-computer data recognition in another path planning method provided in an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of a data recognition model in another path planning method provided according to an embodiment of the present invention;

[0028] Figure 8 is a flowchart of an optional example of another path planning method provided in an embodiment of the present invention;

[0029] Fig. 9 is a structural block diagram of a path planning device provided according to an embodiment of the present invention;

[0030] Fig.10 It is a structural schematic diagram of an electronic device for implementing the path planning method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The situations of "target", "original", etc. are similar and will not be repeated here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 1 is a flow chart of a path planning method provided in an embodiment of the present invention. This embodiment is applicable to the case of path planning. The method can be executed by a path planning device provided in an embodiment of the present invention, which can be implemented by software and / or hardware, and can be integrated in an electronic device, which can be various user terminals or servers.

[0034] See also Figure 1 The method of the embodiment of the present invention specifically includes the following steps:

[0035] S110. In response to a path planning instruction, path planning data is acquired, and the path planning data is identified, and an initial path is generated according to a first identification result obtained, wherein the path planning data includes location data and / or voice data for path planning, and the location data is obtained by collecting the target location.

[0036] The path planning instruction may be an instruction for performing path planning that is automatically or manually triggered, and the path planning data is obtained in response to the path planning instruction. In practical applications, the path planning data may optionally include part data and / or voice data that can be used for path planning. The part data may be data collected from the target part, such as lip shape data collected from the mouth, gesture data collected from the hands, or foot posture data collected from the feet, etc., which are not specifically limited here; the voice data may be data collected from the sound.

[0037] The path planning data is recognized, and an initial path is generated according to a first recognition result obtained thereby. For example, the lip shape data is recognized based on structured light or time of flight (TOF), and an initial path is generated according to a lip shape recognition result obtained thereby; the speech data is recognized based on an acoustic model, and an initial path is generated according to a speech recognition result obtained thereby; etc., which are not specifically limited here.

[0038] S120, obtaining brain-computer data for adjusting the initial path, identifying the brain-computer data, and obtaining path adjustment information according to a second identification result.

[0039] Among them, in actual applications, there may be some differences between the initial path generated based on the above steps and the actual desired moving path for controlling the movement of the mobile robot. In order to plan a target path that better matches the moving path, brain-computer data for adjusting the initial path can be obtained. Optionally, the brain-computer data may include at least one of synchronization potential, desynchronization potential, steady-state visual evoked potential, slow cortical potential, μ rhythm and β rhythm. Further, the brain-computer data is identified, and path adjustment information is obtained based on the second identification result. The path adjustment information is used to indicate how to adjust the initial path, such as moving the end point in the initial path forward by 1 meter; taking the Nth path point in the initial path as the center, rotating the N+1th path point counterclockwise around the center by 30°; etc., which are not specifically limited here.

[0040] It should be noted that brain-computer data is the data formed by the sum of the postsynaptic potentials of a large number of neurons that occur synchronously when the brain is active. It can record the changes in electric waves when the brain is active. It is the overall reflection of the electrophysiological activities of brain nerve cells in the cerebral cortex or scalp surface, that is, it can accurately reflect what humans think. On this basis, considering that mobile robots cannot effectively receive voice commands issued by humans in noisy environments or outer space, but can effectively receive brain-computer commands issued by humans through devices such as brain-computer chips, path adjustment information is generated here by collecting brain-computer data, thereby achieving the effect of effective and accurate generation of path adjustment information in various environments.

[0041] S130: Adjust the initial path based on the path adjustment information to obtain a target path.

[0042] The initial path is adjusted based on the path adjustment information, thereby obtaining a target path that is more in line with the actual desired moving path.

[0043] The technical solution of the embodiment of the present invention obtains path planning data in response to a path planning instruction, identifies the path planning data, and generates an initial path according to the first identification result obtained, wherein the path planning data includes part data and / or voice data for path planning, and the part data is obtained by collecting the target part; obtains brain-computer data for adjusting the initial path, identifies the brain-computer data, and obtains path adjustment information according to the second identification result obtained; adjusts the initial path based on the path adjustment information to obtain the target path. The above technical solution does not need to perform path planning through a pre-set path, and can generate an initial path in real time according to the acquired path planning data, and then generate path adjustment information according to the acquired brain-computer data to adjust the initial path, thereby achieving the effect of flexible path planning, thereby meeting the needs of flexible control of mobile robots in reality.

[0044] Figure 2 It is a flow chart of another path planning method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, when the path planning data includes part data, the path planning data is identified, and an initial path is generated according to the first identification result obtained, which may include: identifying the part data, and generating an initial path according to the first identification result obtained, wherein the initial path is located in the three-dimensional space where the target part is located; after the initial path is generated according to the first identification result obtained, the above-mentioned path planning method also includes: obtaining a target map obtained based on real-time positioning and map construction associated with the three-dimensional space, and matching the initial path to the target map; updating the initial path according to the first matching result obtained. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0045] See also Figure 2 The method of this embodiment may specifically include the following steps:

[0046] S210. In response to a path planning instruction, obtain path planning data, wherein the path planning data may include location data.

[0047] S220, identifying the part data, and generating an initial path according to the obtained first identification result, wherein the initial path is located in the three-dimensional space where the target part is located.

[0048] The part data is identified to generate an initial path in the three-dimensional space where the target part is located.

[0049] S230, obtaining a target map associated with the three-dimensional space based on real-time positioning and map construction, matching the initial path to the target map, and updating the initial path according to the first matching result obtained.

[0050] Among them, the target map may include a map associated with the three-dimensional space based on simultaneous localization and mapping (SLAM), or a SLAM map of the vicinity of the places that the mobile robot passes through during movement. Since the initial path in the three-dimensional space generated based on the part data does not necessarily match the target map, for example, when the target map is reconstructed based on a laser radar and the part data is acquired based on camera collection, the initial path does not match the target map, so the initial path can be matched to the target map to obtain an initial path that matches the target map. In this way, after the target path is subsequently generated based on the initial path, the mobile robot can be controlled to move along the target path that matches the target map.

[0051] S240, obtaining brain-computer data for adjusting the initial path, identifying the brain-computer data, and obtaining path adjustment information according to a second identification result.

[0052] S250: Adjust the initial path based on the path adjustment information to obtain a target path.

[0053] The technical solution of the embodiment of the present invention obtains an initial path in the three-dimensional space where the target part is located by identifying the part data; further, since the initial path does not necessarily match the target map, and the movement process of the mobile robot is controlled based on the target map, the initial path can be matched to the target map, thereby obtaining an initial path that matches the target map.

[0054] An optional technical solution, in the case where the part data includes part videos obtained by capturing the target part at at least two viewing angles, the part data is identified and an initial path is generated based on a first identification result. The solution may include: for the part video captured at each viewing angle, the part video is framed to obtain a part image, and feature extraction is performed on the part image to obtain image features; image features in different part images are matched, and a three-dimensional space is reconstructed based on a second matching result; directional information of the target part is acquired from the three-dimensional space, and based on the directional information, an intersection of an extension direction of the target part and the ground is obtained, and an initial path is generated based on the intersection.

[0055] In order to generate the initial path in the three-dimensional space, the target part is collected at least two viewing angles to obtain a part video. On this basis, the part video collected at each viewing angle can be framed to obtain at least one part image, and then feature extraction is performed on each part image to obtain corresponding image features. Since the part images collected at different viewing angles at the same time have the same feature points, the image features in different part images can be matched, specifically, the image features in different part images collected at the same time can be matched, so as to reconstruct the three-dimensional space. Further, since the reconstructed three-dimensional space includes not only environmental information but also part information of the target part, the directional information of the target part can be obtained from the three-dimensional space. The directional information can be considered as a type of part information, which is used to represent the relevant information of the target part pointing. According to the directional information, the intersection of the extension direction of the target part pointing and the ground can be obtained, and then the initial path can be generated according to these intersections, thereby achieving the effect of effective generation of the initial path.

[0056] In order to understand the above technical solution more vividly, the following is an exemplary description of it with reference to specific examples. Figure 3 Here, taking the target part as an example, the hand is captured by the camera at different viewing angles, and image features (such as scale-invariant feature transform (SIFT)) are extracted from the hand images; then these image features are matched in different hand images, and a set of three-dimensional points and the camera posture are jointly optimized to be consistent with these matches, thereby generating a three-dimensional space. On this basis, the pointing information of the hand is obtained from the three-dimensional space, and the intersection of the extending direction of the hand pointing and the ground is obtained according to the pointing information, so that the initial path can be generated according to these intersections. Finally, the initial path is matched to the SLAM map used to control the movement of the mobile robot.

[0057] In practical applications, the generation (i.e., modeling) process of the above three-dimensional space can be optionally implemented based on the Block-NeRF model. Specifically, the hand image is input into the Block-NeRF model, and then the three-dimensional space is obtained according to the output result of the Block-NeRF model. The specific working process of the above Block-NeRF model is as follows: Figure 4 As shown in the figure: the radiation field and density of the three-dimensional space (i.e., the three-dimensional scene) are modeled based on the Block-NeRF model weights; volume rendering is used to synthesize hand images from perspectives other than the acquired perspective. This synthesis process is through a set of input images of a given three-dimensional scene and its camera pose, and then synthesizes the previously unobserved viewpoints to render the observed scene content.

[0058] Figure 5 It is a flow chart of another path planning method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, the brain-computer data is identified, and the path adjustment information is obtained according to the second identification result obtained, which may include: obtaining a trained brain-computer data recognition model, wherein the brain-computer data recognition model includes a brain-computer feature extraction network and a brain-computer feature recognition network; inputting the brain-computer data into the brain-computer feature extraction network, and obtaining the brain-computer features according to the output results of the brain-computer feature extraction network; inputting the brain-computer features into the brain-computer feature recognition network, and obtaining the path adjustment information according to the second identification result output by the brain-computer feature recognition network. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0059] See also Figure 5 The method of this embodiment may specifically include the following steps:

[0060] S310. In response to a path planning instruction, path planning data is obtained, and the path planning data is identified, and an initial path is generated according to a first identification result obtained, wherein the path planning data includes location data and / or voice data for path planning, and the location data is obtained by collecting the target location.

[0061] S320, obtaining brain-computer data for adjusting the initial path and a trained brain-computer data recognition model, wherein the brain-computer data recognition model includes a brain-computer feature extraction network and a brain-computer feature recognition network.

[0062] Among them, the brain-machine data recognition model is used to identify brain-machine data, which may include a brain-machine feature extraction network for extracting brain-machine features from brain-machine data, and a brain-machine feature recognition network for identifying brain-machine features. In other words, the brain-machine data recognition model includes a brain-machine feature extraction network and a brain-machine feature recognition network connected to each other. In practical applications, optionally, the number of brain-machine brain-machine feature extraction networks can be one, two or more, and each brain-machine feature extraction network can be used to extract brain-machine features in brain-machine data corresponding to a certain or certain path adjustment information. In fact, a path adjustment information can be understood as a path adjustment command, that is, human thought information for adjusting the initial path. Optionally, the above-mentioned brain-machine feature extraction network can be a spiking neural network (Spiking Neuron Networks, SNN) or a Hodgkin-Huxley network, wherein the Hodgkin-Huxley network can be understood as a specific example in SNN.

[0063] S330, inputting brain-computer data into a brain-computer feature extraction network, and obtaining brain-computer features according to output results of the brain-computer feature extraction network.

[0064] S340, inputting the brain-computer feature into the brain-computer feature recognition network, and obtaining path adjustment information according to a second recognition result output by the brain-computer feature recognition network.

[0065] S350: Adjust the initial path based on the path adjustment information to obtain a target path.

[0066] The technical solution of the embodiment of the present invention is to obtain a trained brain-computer data recognition model, which may include a brain-computer feature extraction network and a brain-computer feature recognition network; then, the brain-computer data is input into the brain-computer feature extraction network to obtain brain-computer features, and then the brain-computer features are input into the brain-computer feature recognition network to obtain path adjustment information, thereby achieving the effect of effective recognition of path adjustment information.

[0067] An optional technical solution, the brain-machine feature recognition network may include a generator connected to the brain-machine feature extraction network, and a discriminator connected to the generator; the brain-machine feature is input into the brain-machine feature recognition network, and the path adjustment information is obtained according to the second recognition result output by the brain-machine feature recognition network, which may include: inputting the brain-machine feature into the generator, and obtaining simulation data for simulating brain-machine data according to the output result of the generator; inputting the brain-machine data and the simulation data into the discriminator, and obtaining the path adjustment information according to the second recognition result output by the discriminator. In other words, the brain-machine feature output by the brain-machine feature extraction network can be input into the generator, so that the generator simulates the brain-machine data according to the input brain-machine feature to obtain the simulation data. Further, the brain-machine data and the simulation data are input into the discriminator, so that the discriminator identifies the path adjustment information according to the two.

[0068] In order to understand the above technical solution more vividly, the following is an exemplary description of it with reference to specific examples. Figure 6 As shown, a multi-chamber synaptic connection model and various types of Hodgkin-Huxley models (specific examples of SNN models) are established; a large amount of brain-computer data is collected through the brain-computer chip, and the collected brain-computer data is preprocessed. The brain-computer data may include event-related synchronization potentials, desynchronization potentials, steady-state visual evoked potentials, slow cortical potentials, μ rhythms and β rhythms, which is equivalent to obtaining meta-grid input; the preprocessed brain-computer data is used to train a deep learning model to obtain a data recognition model; the input brain-computer data is recognized based on the data recognition model to obtain path adjustment information (i.e., human thought information). On this basis, for example, see Figure 7 The data recognition model may include an SNN, a generator for capturing data distribution, and a discriminator for estimating the probability that a sample comes from training data. Through an adversarial process, the generator is repeatedly evaluated to determine which model best fits the actual data distribution.

[0069] In order to better understand the above-mentioned technical solutions as a whole, the following is an exemplary description of them in combination with specific examples. Here, a mobile robot represented by an automated guided vehicle (AGV) is taken as an example. Figure 8 The AGV path planning process based on brain-computer chip and gesture recognition is shown in the figure.

[0070] 1. Collect hand videos or gesture videos through the camera.

[0071] 2. Use the Block-NeRF model to recognize the hand video, generate an initial path, then match the initial path to the SLAM map, and update the initial path based on the matching results.

[0072] 3. Collect brain-computer data through brain-computer chips, and identify the brain-computer data through SNN bionic learning algorithm and generative adversarial neural network (generator + discriminator) to obtain human thought information.

[0073] 4. Adjust the initial path on the SLAM map based on human thought information, and then obtain the target path after the detection is completed (that is, without new brain-computer data), so as to navigate and control the AGV.

[0074] 5. In addition, the brain-computer acquisition device can transmit the collected brain-computer data to the controller so that the controller can control the AGV track and speed in real time according to "human thoughts".

[0075] Fig. 9 This is a structural block diagram of a path planning device provided in an embodiment of the present invention, and the device is used to execute the path planning method provided in any of the above embodiments. The device and the path planning method of the above embodiments belong to the same inventive concept, and the details not described in detail in the embodiment of the path planning device can refer to the embodiment of the above path planning method. Fig. 9 The device may specifically include: an initial path generating module 410, a path adjustment information obtaining module 420 and a target path obtaining module 430.

[0076] The initial path generation module 410 is used to obtain path planning data in response to the path planning instruction, identify the path planning data, and generate an initial path according to the first identification result obtained, wherein the path planning data may include part data and / or voice data for path planning, and the part data is obtained by collecting the target part;

[0077] A path adjustment information obtaining module 420 is used to obtain brain-computer data that can be used to adjust the initial path, identify the brain-computer data, and obtain path adjustment information according to the obtained second identification result;

[0078] The target path obtaining module 430 is used to adjust the initial path based on the path adjustment information to obtain the target path.

[0079] Optionally, when the path planning data includes location data, the initial path generation module 410 may include:

[0080] An initial path generating unit, used for identifying the part data, and generating an initial path according to the first identification result, wherein the initial path is located in the three-dimensional space where the target part is located;

[0081] The above-mentioned path planning device may further include:

[0082] An initial path matching module is used to obtain a target map associated with the three-dimensional space and obtained based on real-time positioning and map construction after generating the initial path according to the first recognition result, and match the initial path to the target map;

[0083] The initial path updating module is used to update the initial path according to the obtained first matching result.

[0084] On this basis, optionally, when the part data may include part videos obtained by collecting the target part from at least two viewing angles, the initial path generation unit may include:

[0085] The image feature obtaining subunit is used to divide the part video into frames for each part video collected at each viewing angle to obtain a part image, and to extract features from the part image to obtain image features;

[0086] A three-dimensional space reconstruction subunit, used for matching image features in images of different parts, and reconstructing a three-dimensional space according to the obtained second matching result;

[0087] The initial path generation subunit is used to obtain the directional information of the target part from the three-dimensional space, and obtain the intersection of the extension direction of the target part and the ground according to the directional information, and generate the initial path according to the intersection.

[0088] Alternatively, the target area may include a hand.

[0089] Optionally, the path adjustment information obtaining module 420 may include:

[0090] A brain-computer data recognition model unit, used to obtain a trained brain-computer data recognition model, wherein the brain-computer data recognition model includes a brain-computer feature extraction network and a brain-computer feature recognition network;

[0091] A brain-computer feature obtaining unit, used for inputting brain-computer data into a brain-computer feature extraction network, and obtaining brain-computer features according to an output result of the brain-computer feature extraction network;

[0092] The path adjustment information obtaining unit is used to input the brain-computer feature into the brain-computer feature recognition network, and obtain the path adjustment information according to the second recognition result output by the brain-computer feature recognition network.

[0093] On this basis, optionally, the brain-computer feature recognition network includes a generator connected to the brain-computer feature extraction network, and a discriminator connected to the generator, and the path adjustment information obtaining unit may include:

[0094] A simulation data obtaining subunit is used to input brain-computer features into the generator, and obtain simulation data for simulating brain-computer data according to the output result of the generator;

[0095] The path adjustment information obtaining subunit is used to input brain-computer data and simulation data into the discriminator, and obtain path adjustment information according to the second recognition result output by the discriminator.

[0096] Alternatively, the brain-computer data may include at least one of synchronization potential, desynchronization potential, steady-state visual evoked potential, slow cortical potential, μ rhythm and β rhythm; and / or,

[0097] The brain-machine feature extraction network may include a spiking neural network or a Hodgkin-Huxley network.

[0098] The path planning device provided in the embodiment of the present invention responds to the path planning instruction through the initial path generation module, obtains the path planning data, identifies the path planning data, and generates the initial path according to the first recognition result obtained, the path planning data includes the part data and / or voice data used for path planning, the part data is obtained by collecting the target part; the brain-computer data used to adjust the initial path is obtained through the path adjustment information acquisition module, and the brain-computer data is identified, and the path adjustment information is obtained according to the second recognition result obtained; the initial path is adjusted based on the path adjustment information through the target path acquisition module to obtain the target path. The above device does not need to perform path planning through a pre-set path, and can first generate the initial path in real time according to the acquired path planning data, and then generate the path adjustment information according to the acquired brain-computer data to adjust the initial path, thereby achieving the effect of flexible path planning, thereby meeting the needs of flexible control of mobile robots in reality.

[0099] The path planning device provided in the embodiment of the present invention can execute the path planning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] It is worth noting that in the embodiment of the above-mentioned path planning device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0101] Fig.10 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0102] like Fig.10 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0103] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a path planning method.

[0105] In some embodiments, the path planning method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the path planning method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the path planning method in any other appropriate manner (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0111] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A path planning method, characterized in that: include: In response to a path planning instruction, path planning data is acquired, the path planning data is identified, and an initial path is generated according to a first identification result, wherein the path planning data includes part data and / or voice data for path planning, and the part data is obtained by collecting a target part; the initial path is a path for controlling the movement of the mobile robot; Acquiring brain-computer data for adjusting the initial path, identifying the brain-computer data, and obtaining path adjustment information according to the obtained second identification result; wherein the path adjustment information is used to indicate how to adjust the initial path; Adjusting the initial path based on the path adjustment information to obtain a target path; Wherein, when the path planning data includes the part data, identifying the path planning data and generating an initial path according to the obtained first identification result includes: When the part data includes part videos acquired by capturing the target part at at least two viewing angles, for each of the part videos acquired at the viewing angles, the part videos are framed to obtain part images, and features are extracted from the part images to obtain image features; Matching the image features in the different part images, and reconstructing the three-dimensional space according to the obtained second matching result; Acquire the directional information of the target part from the three-dimensional space, obtain the intersection of the extension direction of the target part and the ground according to the directional information, and generate an initial path according to the intersection; After generating the initial path according to the obtained first recognition result, the method further includes: Acquire a target map associated with the three-dimensional space and obtained based on real-time positioning and map construction, and match the initial path to the target map; The initial path is updated according to the obtained first matching result.

2. The method according to claim 1, characterized in that The target area includes a hand.

3. The method according to claim 1, characterized in that Identifying the brain-computer data and obtaining path adjustment information according to the obtained second identification result, including: Acquire a trained brain-computer data recognition model, wherein the brain-computer data recognition model includes a brain-computer feature extraction network and a brain-computer feature recognition network; Inputting the brain-computer data into the brain-computer feature extraction network, and obtaining brain-computer features according to the output results of the brain-computer feature extraction network; The brain-computer feature is input into the brain-computer feature recognition network, and path adjustment information is obtained according to a second recognition result output by the brain-computer feature recognition network.

4. The method according to claim 3, characterized in that The brain-computer feature recognition network includes a generator connected to the brain-computer feature extraction network, and a discriminator connected to the generator; The step of inputting the brain-computer feature into the brain-computer feature recognition network and obtaining the path adjustment information according to the second recognition result output by the brain-computer feature recognition network includes: Inputting the brain-computer features into the generator, and obtaining simulation data for simulating the brain-computer data according to an output result of the generator; The brain-computer data and the simulation data are input into the discriminator, and path adjustment information is obtained according to a second recognition result output by the discriminator.

5. The method according to claim 3 or 4, characterized in that: The brain-computer data includes at least one of synchronization potential, desynchronization potential, steady-state visual evoked potential, slow cortical potential, μ rhythm and β rhythm; and / or, The brain-computer feature extraction network includes a pulse neural network or a Hodgkin-Huxley network.

6. A path planning device, characterized in that: include: an initial path generation module, for obtaining path planning data in response to a path planning instruction, identifying the path planning data, and generating an initial path according to a first identification result, wherein the path planning data includes part data and / or voice data for path planning, and the part data is obtained by collecting a target part; and the initial path is a path for controlling the movement of the mobile robot; a path adjustment information obtaining module, used to obtain brain-computer data for adjusting the initial path, identify the brain-computer data, and obtain path adjustment information according to the obtained second identification result; wherein the path adjustment information is used to indicate how to adjust the initial path; A target path obtaining module, used for adjusting the initial path based on the path adjustment information to obtain a target path; Among them, the initial path generation module includes: An initial path generating unit, used for identifying the part data, and generating an initial path according to the first identification result, wherein the initial path is located in the three-dimensional space where the target part is located; The device further comprises: An initial path matching module is used to obtain a target map associated with the three-dimensional space and obtained based on real-time positioning and map construction after generating the initial path according to the first recognition result, and match the initial path to the target map; An initial path updating module, used for updating the initial path according to the obtained first matching result; When the part data includes part videos obtained by collecting the target part from at least two viewing angles, the initial path generation unit includes: The image feature obtaining subunit is used to divide the part video into frames for each part video collected at each viewing angle to obtain a part image, and to extract features from the part image to obtain image features; A three-dimensional space reconstruction subunit, used for matching image features in images of different parts, and reconstructing a three-dimensional space according to the obtained second matching result; The initial path generation subunit is used to obtain the directional information of the target part from the three-dimensional space, and obtain the intersection of the extension direction of the target part and the ground according to the directional information, and generate the initial path according to the intersection.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the path planning method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the path planning method as described in any one of claims 1 to 5 when executed.

Images