Information processing method, information processing device, information processing system, and program

Abstract

The present invention realizes a configuration that enables a target landing position for a drone to be determined in accordance with a sentence generated by a user indicating a landing position. The present invention is configured to execute: object identification processing of analyzing a photographed image by a camera attached to a moving body to discriminate an object type of a subject contained in the photographed image; sentence analysis processing of analyzing a sentence indicating a specified landing position for the moving body; word-corresponding subject selection processing of comparing a result of the sentence analysis processing with a result of the object identification processing to select, from the photographed image, a subject corresponding to a word contained in the sentence indicating the specified landing position; and a target landing position determination processing of extracting, from the photographed image, an area corresponding to the specified landing position indicated by the sentence indicating the specified landing position on the basis of a result of the word-corresponding subject selection processing to determine the extracted area as the target landing position for the moving body.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing device, an information processing system, and a program. More specifically, the present disclosure relates to an information processing method, an information processing device, an information processing system, and a program that enable a mobile object such as a drone to land at a location specified by a user. [Background technology]

[0002] In recent years, the use of various autonomous mobile objects such as drones, robots, and self-driving vehicles has been increasing. For example, drones are equipped with cameras and are used to capture images of the ground from above. There are also plans to use drones for package delivery, and various experiments are being conducted.

[0003] Currently, many countries require drones to be controlled under human supervision, i.e., by operating a controller within the range of a person's line of sight. However, in the future, it is expected that autonomous drones that do not require human supervision, i.e., drones that fly autonomously from a departure point to a destination, will be widely used.

[0004] Such autonomous drones fly from their departure point to their destination using, for example, communication information with a control center and GPS location information.

[0005] One specific use case for autonomous drones is parcel delivery, where the user who requested the parcel delivery would be able to receive the parcel by having the drone carrying the parcel land in front of their door. However, it is difficult for the drone to reliably find the user's desired landing location.

[0006] One method for landing a drone at a target location is to use GPS location information, but the latitude and longitude information used as GPS location information often has an error of several meters, making it difficult to reliably land a drone in an extremely small area, such as in front of a house door, using only GPS location information.

[0007] Furthermore, one method for landing a drone at a target location is to record a marker at the target landing location, photograph the marker with a camera attached to the drone, and land by aiming at the marker included in the photographed image. However, having users requesting package delivery create markers that can be seen from the sky places a heavy burden on the users, and is therefore difficult in practice.

[0008] Incidentally, Patent Document 1 (JP 2018-506475 A) is a conventional technology that discloses a configuration for landing a drone at a target position using highly accurate position control. Patent Document 1 discloses a configuration that uses not only GPS position control but also optical sensors, short-range sensors, and radio frequency systems (RF systems) for triangulation to land a drone at a target location with highly accurate position control.

[0009] However, the configuration described in Patent Document 1 is a configuration for accurately landing the drone at the location of a charging system set up for charging the drone, and the landing position is controlled by providing special devices such as a transmitter for guiding the drone to the charging system side, which is the landing position. Therefore, it cannot be applied to landing position control in locations that do not have special facilities, such as a user's garden or front door. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Special Publication No. 2018-506475 Summary of the Invention [Problem to be solved by the invention]

[0011] The present disclosure has been made in consideration of the above-mentioned problems, for example, and aims to provide an information processing method, an information processing device, an information processing system, and a program that enable a mobile object such as a drone to land with high accuracy at a position specified by a user. [Means for solving the problem]

[0012] A first aspect of the present disclosure provides: An information processing method executed in an information processing device, The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; The information processing method executes a target arrival position determination process, which extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0013] Furthermore, a second aspect of the present disclosure is a data processing unit for determining a target arrival position of the moving body; The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; The information processing device executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0014] Furthermore, a third aspect of the present disclosure is An information processing system having a mobile object, a user terminal, and a drone management server, The user terminal transmits a sentence indicating the designated arrival position of the moving object to the drone management server, The drone management server transfers the text indicating the designated arrival position received from the user terminal to the moving body, The moving body, an object identification process for analyzing an image captured by a camera attached to the moving object and determining the object type of the subject included in the captured image; A sentence analysis process that analyzes a sentence indicating a designated arrival location received from the drone management server; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; The information processing system executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0015] Furthermore, a fourth aspect of the present disclosure is An information processing system having a mobile object, a user terminal, and a drone management server, The user terminal transmits a sentence indicating the designated arrival position of the moving object to the drone management server, The moving body transmits an image captured by a camera attached to the moving body to the drone management server, The drone management server, an object identification process for analyzing the captured image received from the moving object and determining the object type of the subject included in the captured image; A sentence analysis process for analyzing a sentence indicating a specified arrival location received from the user terminal; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; extracting an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on a result of the word-corresponding subject selection process, and executing a target arrival position determination process to determine the extracted area as a target arrival position of the moving object; The information processing system transmits the determined target arrival position to the drone.

[0016] Furthermore, a fifth aspect of the present disclosure is An information processing system having a mobile object and a user terminal, a user terminal transmitting a sentence indicating a designated arrival position of the mobile body to the mobile body; The moving body, an object identification process for analyzing an image captured by a camera attached to the moving object and determining the object type of the subject included in the captured image; A sentence analysis process for analyzing a sentence indicating a specified arrival location received from the user terminal; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; The information processing system executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0017] Furthermore, a sixth aspect of the present disclosure is A program for executing information processing in an information processing device, The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; The program executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0018] The program of the present disclosure is a program that can be provided, for example, via a storage medium or communication medium in a computer-readable format to an information processing device or computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or computer system.

[0019] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0020] According to one embodiment of the present disclosure, a configuration is realized that makes it possible to determine the target landing position of a drone according to a sentence indicating the landing position generated by a user. Specifically, for example, the system performs an object identification process that analyzes images captured by a camera attached to the moving body and determines the object type of the subject contained in the captured image; a sentence analysis process that analyzes text indicating the designated arrival position of the moving body; a word-corresponding subject selection process that compares the results of the sentence analysis process with the results of the object identification process and selects from the captured image subjects that correspond to words contained in the text indicating the designated arrival position; and a target arrival position determination process that extracts from the captured image an area that corresponds to the designated arrival position indicated by the text indicating the designated arrival position based on the results of the word-corresponding subject selection process and determines the extracted area as the target arrival position of the moving body. This configuration realizes a configuration that makes it possible to determine the target landing position of the drone according to a sentence indicating the landing position generated by the user. The effects described in this specification are merely examples and are not limiting, and additional effects may also be present. [Brief explanation of the drawings]

[0021] [Figure 1] 10A and 10B are diagrams illustrating an example of a drone landing process using a marker. [Figure 2] 10A and 10B are diagrams illustrating the process of the present disclosure for landing a drone at a designated landing position designated by a user without using a marker. [Figure 3] FIG. 10 is a diagram illustrating the process performed by a user to land a drone at a user-specified landing position. [Figure 4] FIG. 10 is a diagram illustrating the process performed by a user to land a drone at a user-specified landing position. [Figure 5] FIG. 10 is a diagram illustrating an example of a process for transmitting data input by a user to a user terminal such as a smartphone to a drone. [Figure 6] FIG. 10 is a diagram illustrating an example of a process in which data entered by a user into a user terminal such as a smartphone is sent to a drone management server that performs flight control of the drone, etc. [Figure 7] 10A to 10C are diagrams illustrating specific examples of drone flight processing and landing processing to which the processing of the present disclosure is applied. [Figure 8] 10A to 10C are diagrams illustrating specific examples of drone flight processing and landing processing to which the processing of the present disclosure is applied. [Figure 9] FIG. 1 is a diagram illustrating semantic segmentation, which is performed as an object analysis process. [Figure 10] FIG. 10 is a diagram illustrating an example of an object analysis image in which colors according to the object types included in a captured image are output by semantic segmentation. [Figure 11] 10A to 10C are diagrams illustrating specific examples of drone flight processing and landing processing to which the processing of the present disclosure is applied. [Figure 12] 10A and 10B are diagrams illustrating a specific example of morphological analysis processing of a sentence indicating a designated landing position generated by a user. [Figure 13] 10A and 10B are diagrams illustrating a specific example of morphological analysis processing of a sentence indicating a designated landing position generated by a user. [Figure 14] FIG. 10 is a diagram illustrating a specific example of a process for determining to which position in real space the designated position of a sentence, determined by morphological analysis processing of the sentence indicating the designated landing position generated by the user, corresponds. [Figure 15] FIG. 10 is a diagram illustrating a specific example of a process for determining to which position in real space the designated position of a sentence, determined by morphological analysis processing of the sentence indicating the designated landing position generated by the user, corresponds. [Figure 16] FIG. 10 is a diagram illustrating a specific example of a process for determining to which position in real space the designated position of a sentence, determined by morphological analysis processing of the sentence indicating the designated landing position generated by the user, corresponds. [Figure 17] FIG. 10 is a diagram illustrating a specific example of a process for determining to which position in real space the designated position of a sentence, determined by morphological analysis processing of the sentence indicating the designated landing position generated by the user, corresponds. [Figure 18] This figure shows the results of object identification (semantic segmentation) processing performed using dictionary data (trained data) for one object identification. [Figure 19] This figure shows the results of object identification (semantic segmentation) processing performed using dictionary data (trained data) for one object identification. [Figure 20] FIG. 1 is a diagram illustrating a specific example of a Word2Vec dictionary, which is an example of a typical synonym dictionary and enables searching for words that are highly similar to each other. [Figure 21] FIG. 10 is a diagram illustrating an example of a synonym search process using a synonym dictionary. [Figure 22] 10A and 10B are diagrams illustrating a specific example of a process for determining to which position in real space the specified position in a sentence corresponds, using a similar word dictionary for the results of morphological analysis of the sentence indicating the specified landing position. [Figure 23] 10A and 10B are diagrams illustrating a specific example of a process for determining to which position in real space the specified position in a sentence corresponds, using a similar word dictionary for the results of morphological analysis of the sentence indicating the specified landing position. [Figure 24] 10A and 10B are diagrams illustrating a specific example of a process for determining to which position in real space the specified position in a sentence corresponds, using a similar word dictionary for the results of morphological analysis of the sentence indicating the specified landing position. [Figure 25] FIG. 10 is a sequence diagram illustrating the processing sequence of the target landing position determination process of the present disclosure. [Figure 26] FIG. 10 is a sequence diagram illustrating the processing sequence of the target landing position determination process of the present disclosure. [Figure 27] FIG. 10 is a sequence diagram illustrating the processing sequence of the target landing position determination process of the present disclosure. [Figure 28] FIG. 10 is a sequence diagram illustrating the processing sequence of the target landing position determination process of the present disclosure. [Figure 29] FIG. 10 is a sequence diagram illustrating the processing sequence of the target landing position determination process of the present disclosure. [Figure 30] FIG. 10 is a sequence diagram illustrating the processing sequence of the target landing position determination process of the present disclosure. [Figure 31]FIG. 10 is a sequence diagram illustrating the processing sequence of the target landing position determination process of the present disclosure. [Figure 32] FIG. 10 is a sequence diagram illustrating the processing sequence of the target landing position determination process of the present disclosure. [Figure 33] FIG. 2 is a diagram illustrating an example configuration of each device used in the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0022] The information processing method, information processing device, information processing system, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items. 1. Typical examples of drone landing procedures and problems 2. Drone landing position determination process executed by the information processing device of the present disclosure 3. An example of processing when it is difficult to identify an object using object identification processing (semantic segmentation) from a sentence indicating a specified landing location generated by the user. 4. Processing sequence of drone target landing position determination process executed by the information processing device of the present disclosure 4-1. (Processing example 1) A processing example in which user terminal transmission data including designated landing position information (GPS position or address) and text indicating the designated landing position is transmitted from the user terminal to a drone management server, and then the user terminal transmission data is transferred from the drone management server to the drone, and the drone determines the flight path and landing position based on the user terminal transmission data and lands. 4-2. (Processing example 2) A processing example in which user terminal transmission data including designated landing position information (GPS position or address) and text indicating the designated landing position is sent from the user terminal to the drone management server, and then the drone management server determines the drone's flight path and landing position based on the user terminal transmission data, notifies the drone of the determined information, and flies and lands the drone. 4-3. (Processing Example 3) A processing example in which user terminal transmission data including designated landing position information (GPS position or address) and text indicating the designated landing position is transferred from the user terminal to the drone, and the drone determines the flight path and landing position based on the user terminal transmission data and lands. 5. Configuration examples of devices used in the processing of the present disclosure 6. Summary of the Disclosure

[0023] [1. Typical examples of drone landing procedures and problems] First, we will explain a typical example of landing processing for a drone at a target position and the problems that arise.

[0024] As mentioned above, the use of drones has been increasing rapidly in recent years. For example, drones are equipped with cameras to take pictures of the ground from above, and are used to analyze the growth status of agricultural crops on the ground. There are also plans to use drones for package delivery in the future.

[0025] For example, when using drones to deliver packages, an autonomous drone that does not require visual supervision by humans is required, that is, an autonomous drone that flies autonomously from the departure point to the destination.

[0026] Autonomous drones fly from their departure point to their destination, using, for example, communication information with a control center and GPS location information. When delivering packages using autonomous drones, it would be convenient for the user who requested the delivery to have the drone carrying the package land in front of their door and collect it.

[0027] However, as mentioned above, it is difficult for the drone to reliably find the user's desired landing location. For example, when using GPS location information, there is often an error of several meters, making it difficult to land the drone accurately in an extremely narrow area, such as in front of a house door.

[0028] Furthermore, there is a method that uses markers as shown in FIG. As shown in Fig. 1, a marker 20 is recorded at a target landing position for the drone 10. The drone 10 captures an image of the marker 20 with a camera attached to the drone, and lands aiming at the marker included in the captured image. However, having users requesting package delivery create markers that can be seen from the sky, as shown in Figure 1, places a heavy burden on the users, and is therefore difficult in practice.

[0029] The present disclosure solves these problems by enabling a drone to land accurately in a location of a user's choice, such as in the user's garden or in front of the user's front door. The configuration and processing of the present disclosure will be described below.

[0030] [2. Drone landing position determination process executed by the information processing device of the present disclosure] Next, the landing position determination process for a drone executed by the information processing device of the present disclosure will be described.

[0031] The drone landing control process of the present disclosure enables the drone 10 to land accurately in an extremely narrow area, such as in front of a door of a house, as shown in FIG. 2, for example.

[0032] FIG. 2 is a diagram showing an example of a designated landing position 30 designated by a user. Note that Figure 2 shows a dotted oval area as the designated landing location 30, but this is merely an area drawn on the drawing to indicate the location, and there is no dotted line or other shape or marker indicating this area in front of the actual house. That is, the designated landing location 30 shown in dotted lines in the figure is an area that is indistinguishable from other areas of, for example, a front yard of a house.

[0033] For example, a user of a house shown in Figure 2 who has requested a package delivery by drone 10 wishes to have drone 10 land at designated landing location 30 shown in Figure 2, i.e., the area in front of the door of their house, and receive the package delivered by the drone.

[0034] The process that the user performs in such a case will be described with reference to FIG. The user 50 shown in Figure 3 is a resident of the house shown in Figure 2 and wishes to land the drone 10 at the designated landing location 30 described with reference to Figure 2, i.e., the area in front of the door of his or her house.

[0035] In this case, the user 50 inputs the following data into the user terminal 60, such as a smartphone. (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location

[0036] In "(1) Designated landing location (latitude and longitude, or address)", enter the latitude and longitude data or address data corresponding to the designated landing location. If the user 50 is currently at home where the designated landing position is located and the user terminal 60 such as a smartphone has a GPS function, the user can simply input the latitude and longitude data of the current position obtained from the GPS satellite 40. In the example shown in FIG. 3, latitude and longitude data obtained from a GPS satellite 40 is input.

[0037] On the other hand, if the user terminal 60 such as a smartphone does not have a GPS function, or if the current location of the user 50 is different from the designated landing location, the user 50 simply inputs the address of the user's 50 home. The example shown in Figure 4 shows an example where address data is entered in "(1) Designated landing location (latitude and longitude, or address)".

[0038] In the input field for "(2) Sentence indicating designated landing location," the user 50 enters a sentence explaining the location where the user wants to land the drone. For example, as shown in Figures 3 and 4, "The grass near the door" Enter a sentence like this.

[0039] As shown in FIGS. 3 and 4, the user 50 (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location After entering each piece of data, click the send button to send the input data.

[0040] The destination of the data is the drone 10 or a server (drone management server) that manages the flight of the drone 10. The information processing device of the present disclosure that performs the landing position determination process for the drone is configured within the drone 10 or a server that manages the flight of the drone 10 (drone management server).

[0041] FIG. 5 is a diagram showing an example of a process for transmitting data input by a user 50 to a user terminal 60 such as a smartphone to the drone 10. The input data is transmitted to the drone 10 via a communication network. By this data transmission process, the drone 10 receives the user input data, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location Each of these data is received.

[0042] The drone 10 uses this received data to fly to and land at the designated landing location specified by the user 50. The flight and landing processes will be described in detail later.

[0043] On the other hand, FIG. 6 is a diagram showing an example of a process in which data input by a user 50 to a user terminal 60 such as a smartphone is transmitted to a drone management server 70 that performs flight control of the drone 10, etc. The input data is transmitted to the drone management server 70 via a communication network. By this data transmission process, the drone management server 70 receives the user input data, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location Each of these data is received.

[0044] The drone management server 70 uses this received data to generate control signals for flying to and landing at the designated landing position specified by the user 50 and transmits them to the drone 10. The drone 10 performs flying to and landing at the designated landing position specified by the user 50 in accordance with the control signals received from the drone management server 70.

[0045] Alternatively, the drone management server 70 may receive data from the user terminal 60, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location These data may be transferred to the drone 10, which may then analyze the data and fly to and land at a designated landing location. The detailed sequence of these processes will be explained later.

[0046] Next, specific examples of flight processing and landing processing of the drone 10 will be described with reference to Figures 7 and subsequent figures. First, as shown in step S01 of FIG. 7, the drone 10 flies to the vicinity of the designated landing position 30 according to the latitude and longitude information or address information.

[0047] The latitude and longitude information or address information is the user input data described with reference to FIGS. 4 to 6, that is, (1) Designated landing location (latitude and longitude or address) The above data. The drone 10 flies to the vicinity of the designated landing location 30 based on the latitude and longitude information or address data corresponding to the designated landing location 30, which is the user input data.

[0048] When the drone 10 performs autonomous flight, it can analyze its own latitude and longitude position based on, for example, a GPS signal. If the data input by the user is latitude and longitude information, the drone 10 will fly to a location where its own position matches the latitude and longitude information input by the user.

[0049] If the data entered by the user is address data rather than latitude and longitude information, the address is converted to latitude and longitude information using the address / latitude and longitude correspondence data, and then the drone flies to a location where its own latitude and longitude position, analyzed based on the GPS signal, matches the latitude and longitude position corresponding to the converted address.

[0050] The address / latitude and longitude correspondence data is stored and used in a memory unit inside the drone 10. Alternatively, the address / latitude and longitude correspondence data may be stored in a memory unit inside the drone management server 70 that can communicate with the drone 10, and the data stored in the server may be used to analyze the latitude and longitude information corresponding to the address.

[0051] As shown in Figure 7, when the drone 10 flies to the vicinity of the designated landing location 30 based on the latitude and longitude information or address data corresponding to the designated landing location 30, which is user input data, and reaches the sky above the designated landing location 30, the drone 10 then executes the processing of step S02 shown in Figure 8.

[0052] As shown in FIG. 8 (step S02), the drone 10 captures images from the sky using a camera attached to the drone 10, and performs object analysis processing on each of the various subjects included in the captured images.

[0053] Specifically, this object analysis process can be performed using semantic segmentation, for example.

[0054] Semantic segmentation is a type of image recognition process that uses deep learning to identify the object type of the subject in an image, and assigns object-specific pixel values ​​(colors) to the object's constituent pixels according to the identified object type.

[0055] Object identification is performed for each subject in the captured image, and a preset color is assigned according to the identification result. For example, purple is assigned to an identified object if it is a car, red is assigned to a building, etc. Semantic segmentation is a technology that identifies which object category each of the constituent pixels of an image belongs to based on the degree of match between object identification dictionary data (trained data), which registers shape information and other feature information of various actual objects, and the objects in the image captured by a camera.

[0056] This semantic segmentation allows us to identify the types of various objects contained in camera images, such as people, cars, buildings, roads, trees, ponds, grass, etc.

[0057] Figure 9 shows an example of the object identification results using semantic segmentation. Figure 9 shows an example of the results of applying semantic segmentation to an image captured by a camera mounted on a vehicle traveling on a road.

[0058] Through semantic segmentation processing, the pixels that make up the various objects in the captured image are set to pre-defined colors according to the object type and output. That is, the object type of the pixel portion can be determined based on the output color.

[0059] In the example shown in Figure 9, Object type = car, output color = purple Object type = Building, output color = Red Object type = Plant, Output color = Green Object type = Road, output color = Pink Object type = Sidewalk, Output color = Blue Semantic segmentation processing makes it possible to output images with output colors corresponding to such object types.

[0060] As mentioned above, semantic segmentation is a method of identifying which object category each of the constituent pixels of an image belongs to based on the degree of match between the object in the image captured by the camera and dictionary data (trained data) for object identification, which registers shape information and other feature information of various actual objects. For subjects with features not recorded in the dictionary data, the result output is "Object type = Unknown."

[0061] In step S02 shown in Fig. 8, the drone 10 captures images from the sky using a camera attached to the drone 10, and performs object analysis processing on each of the various subjects included in the captured images, i.e., performs object identification processing using semantic segmentation.

[0062] As a result, as shown in FIG. 10, for example, an object analysis image can be acquired in which colors corresponding to the types of various objects included in the photographed image are output. The example shown in Figure 10 is Object type = Roof, output color = Purple Object type = window, output color = blue Object type = Wall, Output color = White Object type = Door, Output color = Red Object type = Tree, output color = Yellow Object type = Grass, Output color = Green FIG. 10 shows an example of outputting colors corresponding to these object types.

[0063] Semantic segmentation processing makes it possible to output images with output colors corresponding to these object types, i.e., semantic segmentation processing makes it possible to distinguish the object types of various subjects contained in an image. After this object identification process, the drone 10 executes the process of step S03 shown in FIG.

[0064] That is, the drone 10 can analyze the position of the door and the position of the lawn based on the results of the semantic segmentation performed as an object identification process as shown in Figure 11, and land accurately on the grass position near the door specified by the user.

[0065] The drone 10 receives the information transmitted from the user terminal 60. Designated landing location text = "The grass near the door" The location corresponding to this sentence (the grass near the door) is determined as the target landing location based on the results of object identification processing (semantic segmentation).

[0066] As a result, a position approximately the same as the designated landing position 30 desired by the user 50 as the drone landing position can be determined as the target landing position, and the drone can land at that position.

[0067] The drone 10 or the drone management server 70 receives the data transmitted from the user terminal 60. Designated landing location text = "The grass near the door" The position specified by this sentence is analyzed to determine which position in the image captured by the camera of drone 10 corresponds to.

[0068] During this process, the drone 10 or the drone management server 70 receives the data transmitted from the user terminal 60, Designated landing location text = "The grass near the door" The text is subjected to morphological analysis or semantic analysis to determine whether the position specified by the text corresponds to a specific position in the image.

[0069] When performing morphological analysis of a sentence, part-of-speech analysis is performed on the morpheme units that make up the sentence, such as nouns and particles contained in the sentence, and then the objects indicated by the nouns and the connections between each noun are analyzed to determine the position specified by the sentence.

[0070] The drone 10 or the drone management server 70 further analyzes, based on the results of the object identification process (semantic segmentation), to determine which position in the image captured by the drone 10's camera corresponds to the position determined by the analysis process of this sentence. The drone 10 sets the position identified based on the analysis results as the target landing position and lands.

[0071] A specific example of the morphological analysis process executed by the drone 10 or the drone management server 70 will be described with reference to Figures 12 and 13. As shown in FIG. 12, it is assumed that the sentence indicating the designated landing position included in the data transmitted from the user terminal 60 is the following sentence: Designated landing location text = "The grass near the door"

[0072] The drone 10 or the drone management server 70 Designated landing location text = "The grass near the door" First, the sentence is separated into morphemes, which are the constituent elements of the sentence, and part-of-speech analysis of each morpheme (noun, particle, etc.) is performed on the morpheme units that make up the sentence.

[0073] As shown in Figure 12, the part of speech of "the grass near the door" is analyzed on a morpheme basis as follows: door = noun no = particle Nearby = noun / adverb no = particle Lawn = noun Note that "nearby" can be set in two ways: as a noun or as an adverb.

[0074] Furthermore, the drone 10 or the drone management server 70 performs a detailed analysis of each morpheme using a morphological analysis dictionary. FIG. 13 is a diagram showing an example of a result of the detailed analysis process of the morphological analysis.

[0075] As shown in Figure 13, the results of analyzing each of the following data on a morpheme-by-morpheme basis are shown: (a) dictionary used, (b) sentence boundary label, (c) written form (surface form), (d) lexeme, (f) lexeme reading, (g) major classification, (h) medium classification, and (i) minor classification. For example, in (b), the sentence boundary label B is a label set for a morpheme at the beginning of a sentence, and I is a label set for a morpheme that is not at the beginning of a sentence. Note that the morphological analysis results shown in FIG. 13 are merely an example, and acquiring all of this data is not essential for the processing of the present disclosure.

[0076] The processing required in the processing of the present disclosure is processing to obtain information that enables determination of the position specified by the sentence sent by the user terminal 60. For this purpose, a part-of-speech analysis is performed on the morpheme units that make up the sentence, such as the nouns and particles contained in the sentence, and the objects indicated by the nouns and the connection relationships between each noun are analyzed to obtain data that enables determination of the position specified by the sentence.

[0077] A specific example of processing for determining to which position in real space a designated position in a sentence determined by morphological analysis processing of the sentence corresponds will be described with reference to FIG. 14 and subsequent figures.

[0078] The process of determining which position in real space the specified position of a sentence determined by the morphological analysis process of the sentence corresponds to is performed based on the results of object identification processing (semantic segmentation) based on images captured by the camera of drone 10.

[0079] Figure 14 shows the following two pieces of data. (A) Morphological analysis results, (B) Object identification (semantic segmentation) results

[0080] First, the drone 10 or the drone management server 70 receives the text indicating the designated landing position from the user terminal 60, i.e., Designated landing location text = "The grass near the door" From the results of morphological analysis of this sentence, morphemes whose part of speech is noun are selected.

[0081] As mentioned above, Designated landing location text = "The grass near the door" The parts of speech of the morphemes contained in this sentence are as follows: door = noun no = particle Nearby = noun / adverb no = particle Lawn = noun

[0082] That is, there are three nouns: "door," "nearby," and "lawn." The drone 10 or the drone management server 70 identifies the objects corresponding to the nouns contained in these sentences as follows: (B) Object identification (semantic segmentation) results It is determined whether the extracted object contains the object.

[0083] As shown in Figure 14(B) Object Identification (Semantic Segmentation) Results, the object identification results are as follows: Roof (output color = purple) Window (Output color = Blue) Wall (output color = white) Door (output color = red) Trees (output color = yellow) Grass (output color = green)

[0084] The drone 10 or the drone management server 70 determines from the object identification result: Sentence = "The grass near the door" The objects corresponding to the nouns in this sentence are: Door (output color = red) Grass (output color = green) These two objects can be detected.

[0085] The drone 10 or the drone management server 70 detects an object corresponding to a noun contained in a sentence detected from the object identification result, i.e., Door (output color = red) Grass (output color = green) Find the location of these two objects, then, Designated landing location text = "The grass near the door" Analyze the real-space position indicated by this sentence. A specific example of processing is shown in FIG.

[0086] As shown in FIG. 15, the drone 10 or the drone management server 70 receives the text indicating the designated landing position from the user terminal 60, i.e., Designated landing location text = "The grass near the door" Analyze the meaning of the morphemes contained in the morphological analysis results of this sentence, Door (output color = red) Grass (output color = green) The morpheme that indicates the relationship between these two objects, i.e. Morpheme = Nearby By analyzing the meaning of the morpheme and the conjunctive relationships of other morphemes such as adverbs, Identify the location in 3D space of the grass near the door.

[0087] As shown in Figure 15, there is a lawn surrounding the house, and the drone 10 or drone management server 70 selects an area of ​​lawn near the door based on text indicating the designated landing location received from the user terminal 60.

[0088] Furthermore, as shown in FIG. 16, the drone 10 or the drone management server 70 determines a grassy area near the door as the target landing position for the drone 10. Through this process, a target landing position 80 shown in FIG. 16 is determined.

[0089] FIG. 17 shows the drone 10 or the drone management server 70 receiving the text indicating the designated landing position transmitted from the user terminal 60, i.e., Designated landing location text = "The grass near the door" The target landing position 80 is determined by morphological analysis, semantic analysis, and object identification processing (semantic segmentation) based on this sentence, Two designated landing positions 30 that have been preset by the user are shown.

[0090] As shown in Figure 17, the target landing position 80 determined by the drone 10 or the drone management server 70 based on the text indicating the designated landing position sent from the user terminal 60 is approximately the same as the designated landing position 30 previously set by the user.

[0091] By landing toward the target landing position 80, the drone 10 can land at approximately the same position as the designated landing position 30 set in advance by the user.

[0092] [3. Example of processing when it is difficult to identify an object using object identification processing (semantic segmentation) from a sentence indicating a designated landing location generated by a user]

[0093] Next, we will explain an example of processing when it is difficult to identify an identified object using object identification processing (semantic segmentation) from a sentence indicating a specified landing position generated by the user.

[0094] As mentioned above, semantic segmentation, which is performed by the drone 10 or the drone management server 70 as an object identification process, is a technology that identifies which object category each of the constituent pixels of an image belongs to based on the degree of match between dictionary data (trained data) for object identification, which registers shape information and other feature information of various actual objects, and the objects in the image captured by the camera.

[0095] This semantic segmentation allows us to identify the types of various objects contained in camera images, such as people, cars, buildings, roads, trees, ponds, grass, etc.

[0096] In this way, semantic segmentation is a process that identifies the object types of various subjects contained in a photographed image using dictionary data (trained data) for object identification, which registers shape information and other feature information of actual objects.Therefore, if the shape or features of the subject in the photographed image to be identified are not registered in the dictionary data (trained data) for object identification, it may not be possible to identify the object for that subject.

[0097] In such a case, simply applying the above-described embodiment may not be enough to determine the target landing position based on the text indicating the designated landing position generated by the user. A process for reducing such a possibility will be described below.

[0098] FIG. 18 shows the results of object identification (semantic segmentation) processing performed using dictionary data (trained data) for one object identification.

[0099] The result of the object identification (semantic segmentation) process shown in Figure 18 is data in which the following multiple objects have been identified: Object = Roof (Output color = Purple) Object = Window (Output Color = Blue) Object = Wall (Output color = White) Object = Door (Output color = Red) Object = Plant (Output color = Green) The colors corresponding to these identified objects are output as an image.

[0100] This object identification result is the result of execution using dictionary data (trained data) for one object identification, and means that the feature information and shape information for each of the identified objects, i.e., roof, window, wall, door, and plant, is registered in the dictionary data (trained data) for object identification.

[0101] However, the object identification result shown in FIG. 18 is different from the object identification result described with reference to FIG. 10 in the previous embodiment. The object identification results described above with reference to FIG. 10 included the following identified objects:

[0102] Object = Roof (Output color = Purple) Object = Window (Output Color = Blue) Object = Wall (Output color = White) Object = Door (Output color = Red) Object = Tree (Output color = Yellow) Object = Grass (Output color = Green)

[0103] That is, the following two types of objects in the object identification result described above with reference to FIG. 10: Object = Tree (Output color = Yellow) Object = Grass (Output color = Green) In the object identification result shown in FIG. 18, these two types of objects are classified into one type of object: Object = Plant (Output color = Green) is set to.

[0104] This is because, in the dictionary data (trained data) used for the object identification process described with reference to Figure 10, i.e., semantic segmentation, feature information and shape information for trees and grass are registered as different objects, whereas in the dictionary data (trained data) used for the object identification process (semantic segmentation) shown in Figure 18, trees and grass are not registered as different objects, but rather feature information and shape information for plants are registered.

[0105] In this way, semantic segmentation processing outputs different object identification results depending on the registered data in the dictionary data (trained data) used in the processing.

[0106] When the above-described embodiment is applied using the object identification result shown in FIG. 18, it becomes impossible to determine the target landing position based on the sentence indicating the designated landing position generated by the user. A specific example will be described with reference to FIG.

[0107] Figure 19 shows the following two pieces of data. (A) Morphological analysis results, (B) Object identification (semantic segmentation) results

[0108] First, the drone 10 or the drone management server 70 receives the text indicating the designated landing position from the user terminal 60, i.e., Designated landing location text = "The grass near the door" From the results of morphological analysis of this sentence, morphemes whose part of speech is noun are selected.

[0109] As mentioned above, Designated landing location text = "The grass near the door" The parts of speech of the morphemes contained in this sentence are as follows: door = noun no = particle Nearby = noun / adverb no = particle Lawn = noun

[0110] That is, there are three nouns: "door," "nearby," and "lawn." The drone 10 or the drone management server 70 identifies the objects corresponding to the nouns contained in these sentences as follows: (B) Object identification (semantic segmentation) results It is determined whether the extracted object contains the object.

[0111] As shown in Figure 19(B) Object Identification (Semantic Segmentation) Results, the object identification results are as follows: Roof (output color = purple) Window (Output color = Blue) Wall (output color = white) Door (output color = red) Plants (output color = green)

[0112] The drone 10 or the drone management server 70 determines from the object identification result: Sentence = "The grass near the door" The objects corresponding to the nouns in this sentence are: Door (output color = red) This one object can be detected.

[0113] However, since the "grass" contained in the sentence is not included in the object identification result shown in FIG. 19(B), it cannot be detected from the object identification result shown in FIG. 19(B).

[0114] In such cases, a synonym dictionary that stores similar data for words is used. The drone 10 or the drone management server 70 stores a dictionary of similar words in its memory, and refers to this dictionary to search for words similar to the words contained in the sentence "the lawn near the door" that indicates the specified landing location generated by the user.

[0115] There are various types of synonym dictionaries, but a typical example is the Word2Vec dictionary. A Word2Vec dictionary is a dictionary that allows you to search for words that are highly similar to each other based on their meaning. It is a dictionary that connects a large number of words using vectors that are set according to the similarity of the meanings of the words. An example of registered data in the Word2Vec dictionary is shown in FIG.

[0116] As shown in FIG. 20, a Word2Vec dictionary has a configuration in which a large number of words are laid out on a two-dimensional xy plane, and the words are connected by vectors according to the similarity between each word. For example, words similar to the word "plant" at the bottom right include "grass," "grass," and "tree."

[0117] The drone 10 or the drone management server 70 performs processing by referring to such a synonym dictionary. i.e., a user-generated statement indicating the desired landing location; Designated landing location text = "The grass near the door" Among the noun words contained in this sentence, for words that are not included in the object identification result, similar words are selected from the similar word dictionary. The similar words to be selected are words that indicate the object included in the object identification result.

[0118] A specific example of the process will be described with reference to FIG. Figure 21 shows (A) Synonym Dictionary (B1) Sentence indicating the designated landing location = "The grass near the door" (B2) Objects registered in the semantic segmentation application dictionary (trained data) Each of these data is shown.

[0119] The drone 10 or the drone management server 70 receives the text indicating the designated landing position generated by the user, i.e., Designated landing location text = "The grass near the door" Among the noun words in this sentence, the words that are not included in the object identification results are: "lawn" to select similar words for "grass" from the thesaurus. The similar words to be selected are words that indicate the object included in the object identification result.

[0120] As shown in the figure, the following words are similar to "grass" and represent objects included in the object identification results: "plant" You can select:

[0121] The drone 10 or the drone management server 70 uses the words selected by applying the thesaurus to convert the sentence indicating the designated landing position received from the user terminal 60, i.e., Designated landing location text = "The grass near the door" A process is performed to determine the intended location of this sentence in 3D space from the results of object identification (semantic segmentation).

[0122] A specific example of the process will be described with reference to FIG. FIG. 22 shows the following two pieces of data. (A) Morphological analysis results, (B) Object identification (semantic segmentation) results

[0123] First, the drone 10 or the drone management server 70 receives the text indicating the designated landing position from the user terminal 60, i.e., Designated landing location text = "The grass near the door" From the results of morphological analysis of this sentence, morphemes whose part of speech is noun are selected.

[0124] As mentioned above, Designated landing location text = "The grass near the door" The parts of speech of the morphemes contained in this sentence are as follows: door = noun no = particle Nearby = noun / adverb no = particle Lawn = noun

[0125] That is, there are three nouns: "door," "nearby," and "lawn." The drone 10 or the drone management server 70 identifies the objects corresponding to the nouns contained in these sentences as follows: (B) Object identification (semantic segmentation) results It is determined whether the extracted object contains the object.

[0126] As shown in Figure 22(B) object identification (semantic segmentation) results, Only the door (output color = red) is detected from the object identification result. "Nearby" and "Grass" are not detected from the object identification results.

[0127] The drone 10 or the drone management server 70 Designated landing location text = "The grass near the door" Among the noun words contained in this sentence, for the noun-corresponding words "nearby" and "lawn" that are not included in the object identification result, similar words are selected from the similar word dictionary. The similar words to be selected are words that indicate the object included in the object identification result.

[0128] In the process of selecting similar words from the similar word dictionary, the drone 10 or the drone management server 70 selects the similar words from the similar word dictionary as described above with reference to FIG. 21 . A word similar to "grass" that indicates an object included in the object identification result. "plant" Select .

[0129] The drone 10 or the drone management server 70 uses the word “plant” selected by applying the thesaurus in this way to convert the sentence indicating the designated landing position received from the user terminal 60, i.e., Designated landing location text = "The grass near the door" A process is performed to determine the intended location of this sentence in 3D space from the results of object identification (semantic segmentation).

[0130] That is, as shown in FIG. A sentence indicating the designated landing position received from the user terminal 60, i.e., Designated landing location text = "The grass near the door" The "grass" in this sentence is determined to be an object that matches "plants," and from "(B) Object Identification Result" shown in Figure 22, Sentence = "The grass near the door" This sentence, "Plants near the door" and the object corresponding to the noun in this interpretation is Door (output color = red) Plants (output color = green) Detect these two objects.

[0131] The drone 10 or the drone management server 70 determines the object detected from the object identification result, i.e., Door (output color = red) Plants (output color = green) Find the location of these two objects, then, Designated landing location text = "The grass near the door" The interpretation of this passage is "Plants near the door" The position in real space indicated by this interpretation sentence is analyzed. A specific processing example is shown in FIG.

[0132] As shown in FIG. 23, the drone 10 or the drone management server 70 receives the text indicating the designated landing position from the user terminal 60, i.e., The sentence indicating the designated landing location = "The grass near the door" This sentence, "Plants near the door" and analyze the meaning of this interpretation.

[0133] in particular, Door (output color = red) Plants (output color = green) The morpheme that indicates the relationship between these two objects, i.e. Morpheme = Nearby By analyzing the meaning of the morpheme and the conjunctive relationships of other morphemes such as adverbs, Identify the location in 3D space of a plant near a door.

[0134] As shown in Figure 23, there are several plants scattered around the house, but the drone 10 or the drone management server 70 selects an area of ​​plants (grass) near the door based on the text indicating the designated landing position received from the user terminal 60.

[0135] Furthermore, as shown in FIG. 24, the drone 10 or the drone management server 70 determines an area of ​​vegetation (grass) near the door as the target landing position for the drone 10. By this process, the target landing position 80 shown in FIG. 24 is determined.

[0136] [4. Processing sequence of drone target landing position determination process executed by the information processing device of the present disclosure] Next, a processing sequence of a target landing position determination process for a drone executed by an information processing device of the present disclosure will be described.

[0137] As mentioned above, the information processing device of the present disclosure that performs the process of determining the landing position of the drone is configured within the drone 10 or the drone management server 70 that manages the flight of the drone 10.

[0138] The sequence diagrams shown in FIG. 25 and subsequent figures are sequence diagrams illustrating the processing sequence of the present disclosure. As processing examples of the present disclosure, the following three types of processing examples will be described in order.

[0139] (Processing example 1) User terminal transmission data including designated landing location information (GPS location or address) and text indicating the designated landing location is transmitted from the user terminal to the drone management server. The user terminal transmission data is then transferred from the drone management server to the drone, and the drone determines the flight path and landing location based on the user terminal transmission data, and lands (Figures 25 to 27).

[0140] (Processing example 2) A processing example in which user terminal transmission data including designated landing location information (GPS location or address) and text indicating the designated landing location is sent from the user terminal to the drone management server, and the drone management server then determines the drone's flight path and landing location based on the user terminal transmission data, notifies the drone of the determined information, and causes the drone to fly and land (Figures 28 to 30).

[0141] (Processing example 3) A processing example in which user terminal transmission data including designated landing position information (GPS position or address) and text indicating the designated landing position is transferred from the user terminal to the drone, and the drone determines the flight path and landing position based on the user terminal transmission data and lands (Figures 31 to 32).

[0142] (4-1. (Processing Example 1) A processing example in which user terminal transmission data including designated landing position information (GPS position or address) and text indicating the designated landing position is transmitted from the user terminal to the drone management server, and then the user terminal transmission data is transferred from the drone management server to the drone, and the drone determines the flight path and landing position based on the user terminal transmission data and lands)

[0143] First, as (Processing Example 1), we will explain a processing example in which user terminal transmission data including designated landing location information (GPS location or address) and text indicating the designated landing location is transmitted from the user terminal to the drone management server, and then the user terminal transmission data is transferred from the drone management server to the drone, and the drone determines the flight path and landing location based on the user terminal transmission data and lands.

[0144] 25 to 27 are sequence diagrams illustrating the above-mentioned (Processing Example 1), which is an example of a sequence of the target landing position determination process of the present disclosure.

[0145] 25 to 27 show, from the left, a user 50, a user terminal 60, a drone management server 70, and a drone 10. As previously described with reference to Figures 3 to 6, user 50 possesses a user terminal 60 such as a smartphone, and inputs GPS location information, address information, and text indicating the designated landing location into user terminal 60.

[0146] The information entered into the user terminal 60 is transmitted to the drone management server 70. The drone management server 70 stores the received information from the user terminal 60 in a memory unit. The drone 10 receives received information from the user terminal 60 that the drone management server 70 stores in the memory unit, analyzes the received information, analyzes the designated landing position desired by the user 50, and determines the target landing position of the drone 10 based on the analysis results.

[0147] The sequence diagrams shown in FIGS. 25 to 27 are sequence diagrams for explaining the communication processing executed between the devices and the processing executed by the devices when these processes are performed. The processing of each step in the sequence diagrams shown in FIGS. 25 to 27 will be explained below in order.

[0148] (Step S101) First, in step S101, the user 50 inputs GPS location information or address information as the designated landing location information of the drone desired by the user 50 to the user terminal 60 such as a smartphone.

[0149] This process corresponds to the process previously described with reference to FIGS. If the user 50 is currently at home where the designated landing position is located and the user terminal 60 such as a smartphone has a GPS function, the user can simply input the latitude and longitude data obtained by the GPS. As explained above with reference to FIG. 3, latitude and longitude data obtained by GPS can be input.

[0150] On the other hand, if the user terminal 60 such as a smartphone does not have a GPS function, or if the current location of the user 50 is different from the designated landing location, the user 50 simply inputs the address of the user's home, as explained with reference to Figure 4.

[0151] (Step S102) Next, in step S102, the user 50 inputs text containing the designated landing position information of the drone desired by the user 50 into the user terminal 60 such as a smartphone.

[0152] This process also corresponds to the process described above with reference to FIGS. The user 50 uses the user terminal 60 shown in FIGS. In the input field for "(2) Text indicating the designated landing position," User 50 enters a sentence describing the desired drone landing location. For example, as shown in Figures 3 and 4, "The grass near the door" Enter a sentence like this.

[0153] (Step S103) Next, in step S103, the user inputs the following to the user terminal 60: (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location Each of these data is transmitted to the drone management server 70.

[0154] As previously described with reference to FIGS. 3 and 4, the user 50 (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location After entering each of these pieces of data, click the send button to send the input data.

[0155] (Step S104) Next, in step S104, the drone management server 70 receives the transmission data of the user terminal 60, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location These pieces of data are received and stored in the storage unit.

[0156] (Step S105) Next, in step S105, the drone 10 that performs a process of delivering a package to the user 50, for example, transmits to the drone management server 70 the data that the drone management server 70 received from the user terminal 60 and stored in the storage unit, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location A transfer request for each of these data is made.

[0157] (Step S106) Next, in step S106, the drone management server 70 transmits the data stored in the storage unit to the drone 10, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location The transmission process for each of these data is executed.

[0158] (Step S107) Next, in step S107, the drone 10 moves (flies) to the vicinity of the designated landing location (latitude and longitude, or address) according to "(1) Designated landing location (latitude and longitude, or address)" received from the drone management server 70.

[0159] (Step S108) Next, in step S108, the drone 10 captures an image of the ground after arriving at the designated landing position (latitude and longitude, or address).

[0160] (Step S109) Next, in step S109, the drone 10 performs an object identification process, such as a semantic segmentation process, based on the captured image.

[0161] As explained above with reference to Figures 9 and 10, semantic segmentation is a technology that identifies which object category each of the constituent pixels of an image belongs to based on the degree of match between object identification dictionary data (trained data), which registers shape information and other feature information of various actual objects, and the objects in the image captured by a camera.

[0162] This semantic segmentation allows us to identify the types of various objects contained in camera images, such as people, cars, buildings, roads, trees, ponds, grass, etc.

[0163] (Step S110) Next, in step S110, the drone 10 performs a matching process between the results of the object identification process (semantic segmentation process) based on the captured image and the sentence indicating the designated landing location desired by the user 50 received from the drone management server 70, detects an object that matches the sentence indicating the designated landing location, and determines the target landing location based on the detected object.

[0164] This process corresponds to the process previously described with reference to FIGS. The drone 10 receives from the drone management server 70 a message indicating the designated landing position desired by the user 50, for example, Designated landing location text = "The grass near the door" It is determined which position in real space the specified position of this sentence corresponds to.

[0165] Specifically, as previously described with reference to FIGS. 12 to 17, Designated landing location text = "The grass near the door" This sentence is separated into morphemes, which are the components of a sentence, and the objects corresponding to the noun morphemes are detected from the results of object identification processing (semantic segmentation processing).

[0166] If an object corresponding to a noun morpheme in a sentence indicating a designated landing location cannot be detected from the results of the object identification process (semantic segmentation process), processing using a synonym dictionary is performed, as previously described with reference to Figures 18 to 24.

[0167] In other words, similar words to the words corresponding to the noun morpheme in the sentence indicating the specified landing location are selected, and the objects corresponding to the selected similar words are detected from the results of the object identification process (semantic segmentation process).

[0168] Furthermore, as described with reference to FIGS. 15 to 17 and 22 to 24, Designated landing location text = "The grass near the door" Analyze the meaning of the morphemes contained in the morphological analysis results of this sentence, The target landing position is determined by analyzing which position in real space the specified position in this sentence corresponds to.

[0169] (Step S111) Finally, in step S111, the drone 10 lands at the target landing position determined in step S110.

[0170] As explained above with reference to Figure 17, the target landing position 80 determined by the drone 10 based on the text indicating the designated landing position sent from the user terminal 60 will be approximately the same as the designated landing position 30 previously set by the user. By landing toward the target landing position 80, the drone 10 can land at approximately the same position as the designated landing position 30 set in advance by the user.

[0171] (4-2. (Processing Example 2) A processing example in which user terminal transmission data including designated landing position information (GPS position or address) and text indicating the designated landing position is sent from the user terminal to the drone management server, and the drone management server then determines the drone's flight path and landing position based on the user terminal transmission data, notifies the drone of the determined information, and causes the drone to fly and land.)

[0172] Next, as (Processing Example 2), we will explain a processing example in which user terminal transmission data including designated landing location information (GPS location or address) and text indicating the designated landing location is transmitted from the user terminal to the drone management server, and then the drone management server determines the drone's flight path and landing location based on the user terminal transmission data, notifies the drone of the determined information, and causes the drone to fly and land.

[0173] 28 to 30 are sequence diagrams illustrating the above-mentioned (Processing Example 2), which is an example of a sequence of the target landing position determination process of the present disclosure.

[0174] 28 to 30 show, from the left, a user 50, a user terminal 60, a drone management server 70, and a drone 10. As previously described with reference to Figures 3 to 6, user 50 possesses a user terminal 60 such as a smartphone, and inputs GPS location information, address information, and text indicating the designated landing location into user terminal 60.

[0175] The information entered into the user terminal 60 is transmitted to the drone management server 70. The drone management server 70 analyzes the information received from the user terminal 60, analyzes the designated landing position desired by the user 50, and determines the flight path and landing position of the drone based on the analysis results. Furthermore, it notifies the drone of the determined information, and causes the drone to fly and land.

[0176] The sequence diagrams shown in FIGS. 28 to 30 are sequence diagrams for explaining the communication processing executed between the devices and the processing executed by the devices when these processes are performed. The processing of each step in the sequence diagrams shown in FIGS. 28 to 30 will be explained below in order.

[0177] (Step S201) First, in step S201, the user 50 inputs GPS location information or address information as the designated landing location information of the drone desired by the user 50 to the user terminal 60 such as a smartphone.

[0178] This process corresponds to the process previously described with reference to FIGS. If the user 50 is currently at home where the designated landing position is located and the user terminal 60 such as a smartphone has a GPS function, the user can simply input the latitude and longitude data obtained by the GPS. As explained above with reference to FIG. 3, latitude and longitude data obtained by GPS can be input.

[0179] On the other hand, if the user terminal 60 such as a smartphone does not have a GPS function, or if the current location of the user 50 is different from the designated landing location, the user 50 simply inputs the address of the user's home, as explained with reference to Figure 4.

[0180] (Step S202) Next, in step S202, the user 50 inputs text containing the designated landing position information of the drone desired by the user 50 into the user terminal 60 such as a smartphone.

[0181] This process also corresponds to the process described above with reference to FIGS. The user 50 uses the user terminal 60 shown in FIGS. In the input field for "(2) Text indicating the designated landing position," User 50 enters a sentence describing the desired drone landing location. For example, as shown in Figures 3 and 4, "The grass near the door" Enter a sentence like this.

[0182] (Step S203) Next, in step S203, the user inputs the following information to the user terminal 60: (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location Each of these data is transmitted to the drone management server 70.

[0183] As previously described with reference to FIGS. 3 and 4, the user 50 (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location After entering each of these pieces of data, click the send button to send the input data.

[0184] (Step S204) Next, in step S204, the drone management server 70 receives the transmission data of the user terminal 60, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location These pieces of data are received and stored in the storage unit.

[0185] (Step S205) Next, in step S205, the drone 10, which performs a process of delivering a package to the user 50, transmits to the drone management server 70 the data that the drone management server 70 received from the user terminal 60 and stored in the storage unit, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location Among these data, a request is made to send "(1) designated landing location (latitude and longitude, or address)."

[0186] (Step S206) Next, in step S206, the drone management server 70 transmits to the drone 10 "(1) Designated landing position (latitude and longitude, or address)" that was received from the user terminal 60 and stored in the memory unit.

[0187] (Step S207) Next, in step S207, the drone 10 moves (flies) to the vicinity of the designated landing location (latitude and longitude, or address) according to "(1) Designated landing location (latitude and longitude, or address)" received from the drone management server 70.

[0188] (Step S208) Next, in step S208, the drone 10 captures an image of the ground after arriving at the designated landing position (latitude and longitude, or address).

[0189] (Step S209) Next, in step S209, the drone 10 transmits to the drone management server 70 an image of the ground captured at the designated landing position (latitude and longitude, or address).

[0190] (Step S210) Next, in step S210, the drone management server 70 performs an object identification process, such as a semantic segmentation process, based on the ground image captured at the designated landing position received from the drone 10.

[0191] As explained above with reference to Figures 9 and 10, semantic segmentation is a technology that identifies which object category each of the constituent pixels of an image belongs to based on the degree of match between object identification dictionary data (trained data), which registers shape information and other feature information of various actual objects, and the objects in the image captured by a camera.

[0192] This semantic segmentation allows us to identify the types of various objects contained in camera images, such as people, cars, buildings, roads, trees, ponds, grass, etc.

[0193] (Step S211) Next, in step S211, the drone management server 70 performs a matching process between the results of the object identification process (semantic segmentation process) based on the captured image and the sentence indicating the designated landing location desired by the user 50 received from the user terminal 60, detects an object that matches the sentence indicating the designated landing location, and determines the target landing location based on the detected object.

[0194] This process corresponds to the process previously described with reference to FIGS. The drone management server 70 receives from the user terminal 60 a sentence indicating the designated landing position desired by the user 50, for example, Designated landing location text = "The grass near the door" It is determined which position in real space the specified position of this sentence corresponds to.

[0195] Specifically, as previously described with reference to FIGS. 12 to 17, Designated landing location text = "The grass near the door" This sentence is separated into morphemes, which are the components of a sentence, and the objects corresponding to the noun morphemes are detected from the results of object identification processing (semantic segmentation processing).

[0196] If an object corresponding to a noun morpheme in a sentence indicating a designated landing location cannot be detected from the results of the object identification process (semantic segmentation process), processing using a synonym dictionary is performed, as previously described with reference to Figures 18 to 24.

[0197] In other words, similar words to the words corresponding to the noun morpheme in the sentence indicating the specified landing location are selected, and the objects corresponding to the selected similar words are detected from the results of the object identification process (semantic segmentation process).

[0198] Furthermore, as described with reference to FIGS. 15 to 17 and 22 to 24, Designated landing location text = "The grass near the door" Analyze the meaning of the morphemes contained in the morphological analysis results of this sentence, The target landing position is determined by analyzing which position in real space the specified position in this sentence corresponds to.

[0199] (Step S212) Next, in step S212, the drone management server 70 transmits the target landing position information determined in step S211 to the drone 10.

[0200] Various processing modes are possible when transmitting target landing position information to the drone 10. Specifically, for example, a process can be applied in which a composite image is generated in which an identification mark indicating the target landing position is set in the captured image received from the drone 10, and the composite image is transmitted to the drone 10.

[0201] (Step S213) Finally, in step S2132, the drone 10 lands at the target landing position according to the target landing position information received from the drone management server 70 in step S212.

[0202] As explained above with reference to Figure 17, the target landing position 80 determined by the drone 10 based on the text indicating the designated landing position sent from the user terminal 60 will be approximately the same as the designated landing position 30 previously set by the user. By landing toward the target landing position 80, the drone 10 can land at approximately the same position as the designated landing position 30 set in advance by the user.

[0203] (4-3. (Processing Example 3) A processing example in which user terminal transmission data including designated landing location information (GPS location or address) and text indicating the designated landing location is transferred from the user terminal to the drone, and the drone determines the flight path and landing location based on the user terminal transmission data and lands)

[0204] Next, as (Processing Example 3), we will explain a processing example in which user terminal transmission data including designated landing location information (GPS location or address) and text indicating the designated landing location is transferred from the user terminal to the drone, and the drone determines the flight path and landing location based on the user terminal transmission data and lands.

[0205] 31 and 32 are sequence diagrams illustrating the above-mentioned (Processing Example 3), which is an example of a sequence of the target landing position determination process of the present disclosure.

[0206] 31 and 32 show, from the left, a user 50, a user terminal 60, and a drone 10. As previously described with reference to Figures 3 to 6, user 50 possesses a user terminal 60 such as a smartphone, and inputs GPS location information, address information, and text indicating the designated landing location into user terminal 60.

[0207] Information entered into the user terminal 60 is transmitted to the drone 10. The drone 10 analyzes the information received from the user terminal 60, analyzes the designated landing position desired by the user 50, and determines the target landing position of the drone 10 based on the analysis results, and lands at that position.

[0208] The sequence diagrams shown in FIGS. 31 and 32 are sequence diagrams for explaining the communication processing executed between the devices and the processing executed by the devices when these processes are performed. The processing of each step in the sequence diagram shown in FIGS. 31 and 32 will be explained below in order.

[0209] (Step S301) First, in step S301, the user 50 inputs GPS location information or address information as the designated landing location information of the drone desired by the user 50 to the user terminal 60 such as a smartphone.

[0210] This process corresponds to the process previously described with reference to FIGS. If the user 50 is currently at home where the designated landing position is located and the user terminal 60 such as a smartphone has a GPS function, the user can simply input the latitude and longitude data obtained by the GPS. As explained above with reference to FIG. 3, latitude and longitude data obtained by GPS can be input.

[0211] On the other hand, if the user terminal 60 such as a smartphone does not have a GPS function, or if the current location of the user 50 is different from the designated landing location, the user 50 simply inputs the address of the user's home, as explained with reference to Figure 4.

[0212] (Step S302) Next, in step S302, the user 50 inputs text containing the designated landing position information of the drone desired by the user 50 into the user terminal 60 such as a smartphone.

[0213] This process also corresponds to the process described above with reference to FIGS. The user 50 uses the user terminal 60 shown in FIGS. In the input field for "(2) Text indicating the designated landing position," User 50 enters a sentence describing the desired drone landing location. For example, as shown in Figures 3 and 4, "The grass near the door" Enter a sentence like this.

[0214] (Step S303) Next, in step S303, the user inputs the following to the user terminal 60: (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location Each of these data is transmitted to the drone 10, which performs the process of delivering the package to the user 50, for example.

[0215] As previously described with reference to FIGS. 3 and 4, the user 50 (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location After entering each of these pieces of data, click the send button to send the input data.

[0216] (Step S304) Next, in step S304, the drone 10 receives the transmission data of the user terminal 60, i.e., (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location These pieces of data are received and stored in the storage unit.

[0217] (Step S305) Next, in step S305, the drone 10 moves (flies) to the vicinity of the designated landing location (latitude and longitude, or address) according to the "(1) designated landing location (latitude and longitude, or address)" received from the user terminal 60.

[0218] (Step S306) Next, in step S306, the drone 10 captures an image of the ground after arriving at the designated landing position (latitude and longitude, or address).

[0219] (Step S307) Next, in step S307, the drone 10 performs an object identification process, such as a semantic segmentation process, based on the captured image.

[0220] As explained above with reference to Figures 9 and 10, semantic segmentation is a technology that identifies which object category each of the constituent pixels of an image belongs to based on the degree of match between object identification dictionary data (trained data), which registers shape information and other feature information of various actual objects, and the objects in the image captured by a camera.

[0221] This semantic segmentation allows us to identify the types of various objects contained in camera images, such as people, cars, buildings, roads, trees, ponds, grass, etc.

[0222] (Step S308) Next, in step S308, the drone 10 performs a matching process between the result of the object identification process (semantic segmentation process) based on the captured image and the sentence indicating the designated landing position desired by the user 50 received from the user terminal 60, detects an object that matches the sentence indicating the designated landing position, and determines the target landing position based on the detected object.

[0223] This process corresponds to the process previously described with reference to FIGS. The drone 10 receives from the user terminal 60 a message indicating the designated landing position desired by the user 50, for example, Designated landing location text = "The grass near the door" It is determined which position in real space the specified position of this sentence corresponds to.

[0224] Specifically, as previously described with reference to FIGS. 12 to 17, Designated landing location text = "The grass near the door" This sentence is separated into morphemes, which are the components of a sentence, and the objects corresponding to the noun morphemes are detected from the results of object identification processing (semantic segmentation processing).

[0225] If an object corresponding to a noun morpheme in a sentence indicating a designated landing location cannot be detected from the results of the object identification process (semantic segmentation process), processing using a synonym dictionary is performed, as previously described with reference to Figures 18 to 24.

[0226] In other words, similar words to the words corresponding to the noun morpheme in the sentence indicating the specified landing location are selected, and the objects corresponding to the selected similar words are detected from the results of the object identification process (semantic segmentation process).

[0227] Furthermore, as described with reference to FIGS. 15 to 17 and 22 to 24, Designated landing location text = "The grass near the door" Analyze the meaning of the morphemes contained in the morphological analysis results of this sentence, The target landing position is determined by analyzing which position in real space the specified position in this sentence corresponds to.

[0228] (Step S309) Finally, in step S309, the drone 10 lands at the target landing position determined in step S308.

[0229] As explained above with reference to Figure 17, the target landing position 80 determined by the drone 10 based on the text indicating the designated landing position sent from the user terminal 60 will be approximately the same as the designated landing position 30 previously set by the user. By landing toward the target landing position 80, the drone 10 can land at approximately the same position as the designated landing position 30 set in advance by the user.

[0230] In the above-described embodiment, an example was described in which a drone was used as a moving body that arrives at a target position, which is a position specified by a user. However, the processing of the present disclosure, i.e., the moving body that arrives at a target position, which is a position specified by a user, is not limited to drones, but can also be applied to various moving bodies other than drones, such as self-driving robots and self-driving vehicles.

[0231] That is, as in the above embodiment, the user generates a sentence indicating the designated arrival location for various moving objects other than drones, such as self-driving robots and self-driving vehicles, and transmits it to the drone 10 or drone management server 70, which is the information processing device of the present disclosure. The drone 10 and the drone management server 70 perform morphological analysis and other analyses of the sentence indicating the specified arrival location, and compare the constituent words of the sentence indicating the specified arrival location with the object identification results based on the image captured by the camera attached to the moving object, thereby analyzing the target arrival location specified by the user.

[0232] In this way, the processing of the present disclosure is effective for making drones and various other mobile objects arrive at an arrival position specified by a user.

[0233] [5. Configuration example of device used for processing of the present disclosure] Next, a configuration example of an apparatus used for the processing of the present disclosure will be described.

[0234] The devices used in the processing of the present disclosure include a drone 10, a user terminal 60, and a drone management server 70. An example of the configuration of each of these devices will be described below.

[0235] FIG. 33 is a diagram showing an example configuration of the drone 10, the user terminal 60, and the drone management server 70. As described above, the information processing device of the present disclosure, i.e., the information processing device that determines the target arrival position, such as the target landing position of a moving body, can be configured in the drone 10 or the drone management server 70. It is also possible to perform processing to determine the target arrival position, such as the target landing position of the moving body, according to the above-described sequence in an information processing device other than the drone 10 or the drone management server 70.

[0236] The user terminal 60 is configured by, for example, a communication terminal such as a smartphone, a device such as a PC, or the like. The drone 10 is not limited to a drone, but may also be a robot, an autonomous vehicle, or the like. The drone management server 70 may be a robot management server, an autonomous vehicle management server, or the like.

[0237] The drone 10, user terminal 60, and drone management server 70 shown in Fig. 33 are configured to be able to communicate with each other. In addition, the drone 10 and user terminal 60 can acquire location information using communication information from GPS satellites.

[0238] As shown in the figure, the drone 10 has a camera 101, a location information acquisition unit 102, a data processing unit 103, a communication unit 104, and a storage unit (memory) 105.

[0239] The camera 101 is used, for example, to capture images of the ground as seen from a drone, or to capture images to be used in self-position estimation processing (for example, SLAM processing).

[0240] The location information acquisition unit 102 communicates with, for example, a GPS satellite, analyzes the current location (latitude, longitude, altitude) of the drone 10 based on the communication information with the GPS satellite, and outputs the analysis information to the data processing unit 103.

[0241] The data processing unit 103 controls the processing executed by the autonomous moving body 100, such as self-localization (SLAM) processing and image capture control. Specifically, the various processes explained in the above-mentioned embodiments are executed. For example, the following process is executed. (a) Flight control for flying according to designated landing position information (latitude, longitude, or address) received from the user terminal 60; (b) analysis of the text indicating the designated landing location received from the user terminal 60; (c) Object analysis processing (semantic segmentation) using images captured by camera 101 (d) Determining the target landing position by matching the sentence indicating the designated landing position received from the user terminal 60 with the result of object analysis processing (semantic segmentation). (e) Landing control to the determined target landing position For example, various processes including these processes are executed.

[0242] The data processing unit 103 has a processor such as a CPU having a program execution function, and executes processing according to a program stored in the storage unit 105 .

[0243] The communication unit 104 performs communication with the user terminal 60 and the drone management server 70. The storage unit (memory) 105 is used as a storage area and a work area for programs executed by the data processing unit 103. It is also used as a storage area for various parameters applied to processing. The storage unit (memory) 105 is composed of RAM, ROM, etc.

[0244] Next, we will explain the configuration of the user terminal 60. As shown in the figure, the user terminal 60 has a camera 201, a position information acquisition unit 202, a data processing unit 203, a storage unit (memory) 204, a communication unit 205, a display unit 206, an input unit 207, and an output unit 208.

[0245] The location information acquisition unit 202 communicates with, for example, a GPS satellite, analyzes the current location (latitude, longitude) of the user terminal 60 based on the communication information from the GPS satellite, and outputs the analysis information to the data processing unit 203.

[0246] The data processing unit 203 executes the various processes described in the above embodiments. for example, (a) Acquisition, display, and transmission of latitude and longitude information as designated landing position information; (b) Display and transmission of address information indicating the designated landing location, which is user-entered information; (c) Display and transmission of text indicating the designated landing position, which is user-entered information; The data processing unit 203 has a processor such as a CPU having a program execution function, and executes processing in accordance with a program stored in the storage unit 204 .

[0247] The storage unit (memory) 204 is used as a storage area and a work area for programs executed by the data processing unit 203. It is also used as a storage area for various parameters applied to processing. The storage unit (memory) 204 is composed of RAM, ROM, etc.

[0248] The communication unit 205 communicates with the drone 10 and the drone management server 70, which is an external server. For example, user input information such as (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location The process of sending this input information to the drone 10 or the drone management server 70 is executed.

[0249] The display unit 206 displays images captured by the camera and also displays user input data, such as (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location The display process of this input information is executed.

[0250] The input unit 207 is a unit operated by the user, and is used for various processes, for example, (1) Designated landing location (latitude and longitude or address) (2) A statement indicating the designated landing location It is used for input processing of this information and input processing of user requests such as data transmission. The output unit 208 includes an audio output unit, an image output unit, and the like.

[0251] Next, a description will be given of the configuration of the drone management server 70. As shown in the figure, the drone management server 70 has a data processing unit 301, a communication unit 302, and a storage unit (memory) 303.

[0252] The data processing unit 301 executes the various processes described in the above embodiments. Specifically, the various processes explained in the above-mentioned embodiments are executed. For example, the following process is executed. (a) generating flight control information for flying according to designated landing position information (latitude, longitude, or address) received from the user terminal 60; (b) analysis of the text indicating the designated landing location received from the user terminal 60; (c) Object analysis processing (semantic segmentation) using images captured by the camera 101 of the drone 100 (d) Determining the target landing position by matching the sentence indicating the designated landing position received from the user terminal 60 with the result of object analysis processing (semantic segmentation). (e) Processing to generate landing control information for the determined target landing position For example, various processes including these processes are executed.

[0253] The data processing unit 301 has a processor such as a CPU having a program execution function, and executes processing in accordance with a program stored in the storage unit 303 .

[0254] The communication unit 302 performs communication with the drone 10 and the user terminal 60. The storage unit (memory) 303 is used as a storage area and a work area for programs executed by the data processing unit 301. It is also used as a storage area for various parameters applied to processing. The storage unit (memory) 303 is composed of RAM, ROM, etc.

[0255] As mentioned above, in the above-described embodiments, an example was described in which the moving body was a drone, but the processing of the present disclosure is not limited to drones and can also be applied to other moving bodies, such as robots and autonomous vehicles. Similar processing can be performed by replacing the drones in the above-described embodiments with robots or autonomous vehicles.

[0256] 6. Summary of the Disclosure The embodiments of the present disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0257] The technology disclosed in this specification can be configured as follows. (1) An information processing method executed in an information processing device, The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; An information processing method that executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by a sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0258] (2) The moving object is a drone, The target arrival position of the moving body is a target landing position of the drone, The data processing unit In the target arrival position determination process, The information processing method described in (1) determines a target landing position for a drone.

[0259] (3) The text indicating the designated arrival location is a text transmitted from an external terminal, The data processing unit The information processing method according to (1) or (2), wherein the text analysis process executes an analysis process of a text indicating a designated arrival position transmitted from the external terminal.

[0260] (4) The text indicating the designated arrival location is a text input into a user terminal by a user requesting the movement of the moving object and transmitted from the user terminal, The data processing unit The information processing method according to any one of (1) to (3), wherein the text analysis process executes an analysis process of a text indicating a designated arrival position transmitted from the user terminal.

[0261] (5) The data processing unit In the sentence analysis process, a morphological analysis of the sentence indicating the specified arrival position is performed; The information processing method according to any one of (1) to (4), wherein in the word-corresponding object selection process, an object corresponding to the word extracted by the morphological analysis is selected from the captured image.

[0262] (6) The data processing unit The information processing method according to (5), further comprising selecting, from the captured image, a subject corresponding to a noun among the words analyzed by the morphological analysis.

[0263] (7) The data processing unit Sent from the user terminal, (a) latitude and longitude information or address information of the designated arrival location; (b) a statement indicating said designated arrival location; Enter the above data (a) and (b), The information processing method according to any one of (1) to (6), further comprising determining a moving route and a target arrival position of the moving body.

[0264] (8) The data processing unit The information processing method according to any one of (1) to (7), wherein semantic segmentation is performed in the object identification process.

[0265] (9) The data processing unit In the word-corresponding object selection process, If a subject corresponding to a word included in the sentence indicating the specified arrival position cannot be selected from the captured image, An information processing method described in any one of (1) to (8), which performs a process of searching for similar words similar to words contained in a sentence indicating the specified arrival location and selecting subjects corresponding to the similar words from the captured image.

[0266] (10) The information processing device is configured inside the moving body, The data processing unit The information processing method according to any one of (1) to (9), wherein the text to be analyzed in the text analysis process is received from a user terminal or an external server.

[0267] (11) The information processing device is configured inside the mobile body, The data processing unit The information processing method according to any one of (1) to (10), further comprising: executing a movement control for moving the moving device toward the target arrival position determined in the target arrival position determination process.

[0268] (12) The information processing device is configured in a server capable of communicating with the mobile object, The data processing unit The information processing method according to any one of (1) to (11), wherein the text to be analyzed in the text analysis process is received from a user terminal.

[0269] (13) The information processing device is configured in a server capable of communicating with the mobile object, The data processing unit The information processing method according to any one of (1) to (12), wherein a process is executed to transmit the target arrival position determined in the target arrival position determination process to the mobile device.

[0270] (14) A data processing unit for determining a target arrival position of the moving body; The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; An information processing device that executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0271] (15) An information processing system having a mobile object, a user terminal, and a drone management server, The user terminal transmits a sentence indicating the designated arrival position of the moving object to the drone management server, The drone management server transfers the text indicating the designated arrival position received from the user terminal to the moving body, The moving body, an object identification process for analyzing an image captured by a camera attached to the moving object and determining the object type of the subject included in the captured image; A sentence analysis process that analyzes a sentence indicating a designated arrival location received from the drone management server; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; An information processing system that executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0272] (16) An information processing system having a mobile object, a user terminal, and a drone management server, The user terminal transmits a sentence indicating the designated arrival position of the moving object to the drone management server, The moving body transmits an image captured by a camera attached to the moving body to the drone management server, The drone management server, an object identification process for analyzing the captured image received from the moving object and determining the object type of the subject included in the captured image; A sentence analysis process for analyzing a sentence indicating a specified arrival location received from the user terminal; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; extracting an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on a result of the word-corresponding subject selection process, and executing a target arrival position determination process to determine the extracted area as a target arrival position of the moving object; An information processing system that transmits the determined target arrival position to the drone.

[0273] (17) An information processing system having a mobile object and a user terminal, a user terminal transmitting a sentence indicating a designated arrival position of the mobile body to the mobile body; The moving body, an object identification process for analyzing an image captured by a camera attached to the moving object and determining the object type of the subject included in the captured image; A sentence analysis process for analyzing a sentence indicating a specified arrival location received from the user terminal; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; An information processing system that executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0274] (18) A program for executing information processing in an information processing device, The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; A program that executes a target arrival position determination process that extracts an area from the captured image that corresponds to the designated arrival position indicated by the sentence indicating the designated arrival position based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

[0275] Furthermore, the series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program on a computer from the recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0276] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are all located in the same housing. [Industrial Applicability]

[0277] As described above, according to the configuration of one embodiment of the present disclosure, a configuration is realized that makes it possible to determine the target landing position of a drone in accordance with a sentence indicating the landing position generated by the user. Specifically, for example, the system performs an object identification process that analyzes images captured by a camera attached to the moving body and determines the object type of the subject contained in the captured image; a sentence analysis process that analyzes text indicating the designated arrival position of the moving body; a word-corresponding subject selection process that compares the results of the sentence analysis process with the results of the object identification process and selects from the captured image subjects that correspond to words contained in the text indicating the designated arrival position; and a target arrival position determination process that extracts from the captured image an area that corresponds to the designated arrival position indicated by the text indicating the designated arrival position based on the results of the word-corresponding subject selection process and determines the extracted area as the target arrival position of the moving body. This configuration realizes a configuration that makes it possible to determine the target landing position of the drone according to a sentence indicating the landing position generated by the user. [Explanation of symbols]

[0278] 10. Drone 30 Designated landing position 40 GPS satellites 50 users 60 User terminals 70 Drone Management Server 101 Camera 102 Location information acquisition unit 103 Data Processing Unit 104 Communications Department 105 Memory 201 Camera 202 Location information acquisition section 203 Data Processing Unit 204 Memory 205 Communications Department 206 Display section 207 Input section 208 Output section 301 Data Processing Unit 302 Communications Department 303 Memory

Claims

1. An information processing method executed in an information processing device, The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; An information processing method that executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by a sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

2. the moving object is a drone, The target arrival position of the moving body is a target landing position of the drone, The data processing unit In the target arrival position determination process, The information processing method of claim 1 , wherein a target landing position of a drone is determined.

3. the text indicating the designated arrival location is a text transmitted from an external terminal, The data processing unit 2. The information processing method according to claim 1, wherein the text analysis process includes analyzing a text indicating a designated arrival location transmitted from the external terminal.

4. the sentence indicating the designated arrival position is a sentence input into a user terminal by a user requesting the movement of the moving object and transmitted from the user terminal, The data processing unit 2. The information processing method according to claim 1, wherein the text analysis process includes analyzing a text indicating a designated arrival location transmitted from the user terminal.

5. The data processing unit In the sentence analysis process, a morphological analysis of the sentence indicating the specified arrival position is performed; 2. The information processing method according to claim 1, wherein in the word-corresponding object selection process, an object corresponding to the word extracted by the morphological analysis is selected from the captured image.

6. The data processing unit 6. The information processing method according to claim 5, wherein a subject corresponding to a noun among the words analyzed by the morphological analysis is selected from the photographed image.

7. The data processing unit Sent from the user terminal, (a) latitude and longitude information or address information of the designated arrival location; (b) a statement indicating the designated arrival location; Enter the above data (a) and (b), The information processing method according to claim 1, wherein a moving route and a target arrival position of said mobile object are determined.

8. The data processing unit The information processing method according to claim 1 , wherein semantic segmentation is performed in the object identification process.

9. The data processing unit In the word-corresponding object selection process, If a subject corresponding to a word included in the sentence indicating the specified arrival location cannot be selected from the captured image, The information processing method according to claim 1 , further comprising the steps of: searching for words similar to words contained in the sentence indicating the specified arrival position; and selecting, from the captured image, a subject corresponding to the similar word.

10. the information processing device is configured inside the mobile object, The data processing unit 2. The information processing method according to claim 1, wherein the text to be analyzed in the text analysis process is received from a user terminal or an external server.

11. the information processing device is configured inside the mobile object, The data processing unit The information processing method according to claim 1 , further comprising the step of executing a movement control for moving the mobile object toward the target arrival position determined in the target arrival position determination process.

12. the information processing device is configured inside a server capable of communicating with the mobile object, The data processing unit The information processing method according to claim 1 , wherein the text to be analyzed in the text analysis process is received from a user terminal.

13. the information processing device is configured inside a server capable of communicating with the mobile object, The data processing unit 2. The information processing method according to claim 1, further comprising the step of transmitting the target arrival position determined in the target arrival position determination process to the mobile unit.

14. a data processing unit for determining a target arrival position of the moving body; The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; An information processing device that executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

15. An information processing system having a mobile object, a user terminal, and a drone management server, The user terminal transmits a sentence indicating the designated arrival position of the moving object to the drone management server, The drone management server transfers the text indicating the designated arrival position received from the user terminal to the moving body, The moving body, an object identification process for analyzing an image captured by a camera attached to the moving object and determining the object type of the subject included in the captured image; A sentence analysis process that analyzes a sentence indicating a designated arrival location received from the drone management server; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; An information processing system that executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

16. An information processing system having a mobile object, a user terminal, and a drone management server, The user terminal transmits a sentence indicating the designated arrival position of the moving object to the drone management server, The moving body transmits an image captured by a camera attached to the moving body to the drone management server, The drone management server, an object identification process for analyzing the captured image received from the moving object and determining the object type of the subject included in the captured image; A sentence analysis process for analyzing a sentence indicating a specified arrival location received from the user terminal; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; extracting an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on a result of the word-corresponding object selection process, and executing a target arrival position determination process to determine the extracted area as a target arrival position of the moving object; An information processing system that transmits the determined target arrival position to the mobile unit.

17. An information processing system having a mobile object and a user terminal, a user terminal transmitting a sentence indicating a designated arrival position of the mobile body to the mobile body; The moving body, an object identification process for analyzing an image captured by a camera attached to the moving object and determining the object type of the subject included in the captured image; A sentence analysis process for analyzing a sentence indicating a specified arrival location received from the user terminal; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; An information processing system that executes a target arrival position determination process that extracts an area corresponding to the designated arrival position indicated by the sentence indicating the designated arrival position from the captured image based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

18. A program for executing information processing in an information processing device, The data processing unit an object identification process for analyzing an image captured by a camera attached to a moving object and determining the object type of the subject included in the captured image; a sentence analysis process for analyzing a sentence indicating a designated arrival position of the moving object; a word-corresponding object selection process for comparing a result of the sentence analysis process with a result of the object identification process and selecting, from the captured image, an object corresponding to a word included in the sentence indicating the specified arrival position; A program that executes a target arrival position determination process that extracts an area from the captured image that corresponds to the designated arrival position indicated by the sentence indicating the designated arrival position based on the result of the word-corresponding subject selection process, and determines the extracted area as the target arrival position of the moving body.

Images