Information processing method, program, and information processing device

Abstract

To provide an information processing method and the like capable of improving identification accuracy of an object.SOLUTION: An information processing method includes: acquiring image information on an image of an object captured by an information processing terminal; acquiring sensor information detected by a sensor of the information processing terminal when capturing the object; and referring to a storage section that stores the image information and the sensor information corresponding to a plurality of object candidates to execute processing of identifying the captured object from the plurality of object candidates on the basis of the acquired image information and sensor information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing method, a program, and an information processing apparatus.

Background Art

[0002] In recent years, the need to identify objects based on images has been increasing. Patent Document 1 discloses an identification method in which an identifier corresponding to the shooting conditions is selected from a plurality of identifiers generated for each of a plurality of different shooting conditions, and the object in the image data is identified using the selected identifier.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the invention according to Patent Document 1 has a problem that it cannot identify (specify) an object when the photographed objects have the same shape, when there is no change in the background of the object, or when there are a plurality of objects having the same shape.

[0005] On one aspect, an object is to provide an information processing method or the like capable of improving the identification accuracy of an object.

Means for Solving the Problems

[0006] An information processing method according to one aspect acquires image information regarding an image of an object photographed by an information processing terminal, The information processing terminal includes a plurality of sensors, at the time of photographing the object, the each sensor information detected by a plurality of sensor of the information processing terminal is acquired, obtains each evaluation item related to the acquired image information and a plurality of sensor information, and the weighting corresponding to each evaluation item, and based on each obtained evaluation item and the weighting corresponding to each evaluation item, and a storage unit that stores image information and sensor information corresponding to a plurality of object candidates for each object candidate stored therein, calculates the similarity between the object and the object candidate, and selects the object candidate with the highest similarity as The photographed object as It is characterized by causing a process for specifying to be executed.

Effect of the Invention

[0007] On one side, it becomes possible to improve the identification accuracy of the object.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0009] Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments.

[0010] (Embodiment 1) Embodiment 1 relates to a form of identifying an object captured from among a plurality of object candidates based on image information of the object captured by an information processing terminal and sensor information detected by a sensor of the information processing terminal. The object is, for example, a power receiving and transforming facility such as a switchboard or a distribution board, an environmental measurement device, any device such as a personal computer, a vehicle, an object, an animal, or a person.

[0011] FIG. 1 is an explanatory diagram showing an overview of an object identification system. The system of this embodiment includes an information processing device 1 and an information processing terminal 2, and each device transmits and receives information via a network N such as the Internet.

[0012] The information processing device 1 is an information processing device that performs processing, storage, and transmission / reception of various information. The information processing device 1 is, for example, a server device, a personal computer, or a general-purpose tablet PC (personal computer), etc. In this embodiment, the information processing device 1 is assumed to be a server device, and hereinafter it will be read as server 1 for simplicity.

[0013] The information processing terminal 2 is a terminal device that acquires image information related to an image of an object, detects sensor information by a sensor of the information processing terminal 2, and transmits the image information and the sensor information, etc. The information processing terminal 2 is, for example, an information processing device such as a smartphone, a mobile phone, a wearable device such as an Apple Watch (registered trademark), a tablet, wearable glasses, or a personal computer terminal. Hereinafter, for simplicity, the information processing terminal 2 will be read as terminal 2.

[0014] The server 1 according to this embodiment acquires image information regarding an image of an object photographed by a terminal 2 provided with a plurality of sensors. The server 1 acquires a plurality of sensor information detected by each sensor of the terminal 2 at the time of photographing the object. The server 1 refers to a storage unit that stores image information and sensor information corresponding to a plurality of object candidates, and calculates the similarity between the object and each object candidate based on the acquired image information and the plurality of sensor information. The server 1 specifies the object candidate with the highest similarity as the photographed object. Note that the image information and the sensor information will be described later.

[0015] FIG. 2 is a block diagram showing a configuration example of the server 1. The server 1 includes a control unit 11, a storage unit 12, a communication unit 13, a reading unit 14, and a mass storage unit 15. Each component is connected by a bus B.

[0016] The control unit 11 includes an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit), and reads and executes a control program 1P (program product) stored in the storage unit 12, thereby performing various information processes, control processes, etc. related to the server 1. Note that the control program 1P can be deployed to be executed on a single computer, or arranged at one site, or distributed over a plurality of sites and executed on a plurality of computers interconnected by a communication network. In FIG. 2, the control unit 11 is described as a single processor, but it may be a multi-processor.

[0017] The storage unit 12 includes memory elements such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and stores a control program 1P, data, etc. necessary for the control unit 11 to execute processing. Further, the storage unit 12 temporarily stores data, etc. necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information to and from the terminal 2, etc. via the network N.

[0018] The reading unit 14 reads a portable storage medium la including a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The control program 1P may be read from the portable storage medium la by the control unit 11 via the reading unit 14 and stored in the large-capacity storage unit 15. Further, the control program 1P may be downloaded by the control unit 11 from another computer via the network N, etc. and stored in the large-capacity storage unit 15. Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

[0019] The large-capacity storage unit 15 includes a recording medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The large-capacity storage unit 15 includes an object DB (database) 151. The object DB 151 stores information about the object, image information of the object, sensor information detected by the sensor of the terminal 2 at the time of photographing the object, etc.

[0020] Note that in this embodiment, the storage unit 12 and the large-capacity storage unit 15 may be configured as an integrated storage device. Further, the large-capacity storage unit 15 may be configured by a plurality of storage devices. Furthermore, the large-capacity storage unit 15 may be an external storage device connected to the server 1.

[0021] Server 1 may execute various information processing and control processing, etc. either by a single computer or in a distributed manner by a plurality of computers. Further, Server 1 may be realized by a plurality of virtual machines provided in one server, or may be realized using a cloud server. Note that the processing of Server 1 may be executed on Terminal 2.

[0022] FIG. 3 is an explanatory diagram showing an example of the record layout of the object DB 151. The object DB 151 includes an object ID (Identifier) column, a type column, an installation information column, an image information column, and a sensor information column. The object ID column stores the ID of the object uniquely specified to identify each object. The type column stores the type of the object (such as a switchboard or a distribution board). The installation information column stores owner information such as the name of the person who owns the object, or the installation position, etc.

[0023] The image information column includes a character string column, a feature point column, and a feature amount column. The character string column stores the characters included in the image of the object and the coordinates of the characters, etc. The feature points store the feature points included in the image of the object and the coordinates of the feature points, etc. The feature points are, for example, the blanking shape of a press or a drill hole, an edge shape, paint, a seal, a forced indentation hole, a laser imprint, or three-dimensional characters, etc.

[0024] The feature amount column stores the feature amount data of the image of the object. The feature amount data includes RGB (Red, Blue, Green) gradation values, YMCK (Yellow, Cyan, Magenta, Key plate) gradation values, lightness, luminance, hue, density, chroma or contrast-related feature amounts, or LBP (Local Binary Pattern) feature amounts, etc. The LBP feature amount is composed of relative values obtained by comparing each pixel with neighboring pixels in the vicinity as feature amounts of the local expression of the image.

[0025] The sensor information column includes a wireless LAN column, a GPS (Global Positioning System, Global Positioning Satellite) column, an air pressure column, an altitude column, a gyro sensor column, and a LiDAR (Light Detection and Ranging) sensor column. The wireless LAN column stores the wireless LAN information of terminal 2. Note that the wireless LAN information will be described later. The GPS column stores the GPS data (latitude and longitude, etc.) read via the GPS function of terminal 2.

[0026] The air pressure column stores the air pressure data acquired by a pressure sensor (barometer) built into terminal 2. The altitude column stores the altitude data acquired by an altitude sensor (altimeter) built into terminal 2. The gyro sensor column stores the gyro sensor data acquired by a gyro sensor (angular velocity sensor) built into terminal 2. A gyro sensor is a type of inertial sensor that realizes the measurement of rotational angular velocity (the speed at which an object is rotating).

[0027] The LiDAR sensor column stores the LiDAR sensor data acquired by a LiDAR sensor built into terminal 2. A LiDAR sensor irradiates an object while scanning a laser beam and observes the scattered or reflected light, thereby detecting the distance, position, shape, etc. of an object located at a distance.

[0028] Note that the storage form of each of the above-mentioned databases is an example, and other storage forms may be used as long as the relationship between the data is maintained.

[0029] Figure 4 is a block diagram showing a configuration example of terminal 2. Terminal 2 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, a display unit 25, a photographing unit 26, a GPS module 27, an air pressure module 28, an altitude module 29, a gyro module 30, and a LiDAR module 31. Each component is connected by a bus B.

[0030] The control unit 21 includes an arithmetic processing unit such as a CPU or an MPU, and performs various information processing, control processing, etc. related to the terminal 2 by reading and executing a control program 2P (program product) stored in the storage unit 22. In FIG. 4, the control unit 21 is described as a single processor, but it may also be a multi-processor. The storage unit 22 includes memory elements such as a RAM and a ROM, and stores a control program 2P, data, etc. necessary for the control unit 21 to execute processing. Further, the storage unit 22 temporarily stores data, etc. necessary for the control unit 21 to execute arithmetic processing.

[0031] The communication unit 23 is a communication module for performing communication-related processing, and transmits and receives information to and from the server 1 etc. via the network N. The input unit 24 may be a keyboard, a mouse, or a touch panel integrated with the display unit 25. The display unit 25 is a liquid crystal display, an organic EL (electroluminescence) display, etc., and displays various information according to an instruction from the control unit 21.

[0032] The imaging unit 26 is an imaging device such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera. Note that the imaging unit 26 may not be built into the terminal 2, but may be directly connected to the terminal 2 externally and be configured to be able to take pictures.

[0033] The GPS module 27 is a module for acquiring position information (latitude and longitude, etc.) using GPS satellites. The pressure module 28 is a sensor module for acquiring pressure data. The altitude module 29 is a sensor module for acquiring altitude data. The gyro module 30 is a sensor module for acquiring gyro sensor data. The lidar module 31 is a sensor module for acquiring lidar sensor data. Note that it is not limited to the types of sensor modules described above, and for example, an acceleration module for acquiring acceleration sensor data may be provided.

[0034] Note that the pressure module 28, altitude module 29, gyro module 30, or rider module 31 may not be built into the terminal 2, but may be directly connected to the terminal 2 externally and configured to be able to acquire sensor data.

[0035] Note that a combined use of a plurality of types among the pressure module 28, altitude module 29, gyro module 30, and rider module 31 may also have an integrated structure.

[0036] FIG. 5 is an explanatory diagram showing a processing operation for identifying an object photographed by the terminal 2. The terminal 2 photographs an object via the photographing unit 26. The terminal 2 acquires image information regarding the image of the photographed object. The image information includes characters or feature points included in the image of the object, or feature amounts of the image.

[0037] Regarding the feature amounts of the image, the terminal 2 extracts the feature amounts of the image of the photographed object using a local feature amount extraction method such as A-KAZE (Accelerated KAZE), SIFT (Scale Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), or HOG (Histograms of Oriented Gradients). Regarding the characters or feature points, for example, the terminal 2 extracts the feature amounts of the image of the object and recognizes the characters or feature points in the photographed image based on the extracted feature amounts. Note that the characters or feature points may be automatically recognized using a learning model constructed by machine learning, or may be recognized using OpenCV (Open Source Computer Vision Library), which is an image processing library.

[0038] The terminal 2 acquires sensor information detected by the sensors of the terminal 2 when photographing the object. The sensor information includes wireless LAN information, satellite positioning system data (e.g., GPS data), barometric pressure data, altitude data, gyro sensor data, or lidar sensor data. Note that it is not limited to these sensor data. For example, the sensor information may include AR (Augmented Reality) data, MR (Mixed Reality) data, geomagnetic (compass) sensor data, acceleration sensor data, luminance sensor data, proximity sensor data, or fingerprint sensor data. That is, depending on the location, environment, situation, or state, etc. when photographing the object, sensor data detected by various types of sensors may be used.

[0039] The terminal 2 acquires wireless LAN information using a wireless communication method. The wireless LAN information may include at least one of the MAC address (Media Access Control address) of the access point, the SSID (Service Set Identifier) which is an identifier of the wireless LAN function, the radio wave intensity of the SSID, and the frequency channel. The wireless communication method includes a BLE (Bluetooth Low Energy) communication method, a communication method compliant with the Bluetooth (registered trademark) standard other than the BLE standard, a communication method compliant with a short-range wireless communication standard other than the Bluetooth standard, a Wi-Fi (registered trademark) Direct communication method, or a terminal wireless communication method such as LTE (Long Term Evolution), 4G (Generation), or 5G.

[0040] The terminal 2 receives transmission radio waves from GPS positioning satellites via the GPS function of the terminal 2 to acquire GPS data (latitude and longitude, etc.). The terminal 2 acquires barometric pressure data, altitude data, gyro sensor data, and lidar sensor data respectively from the pressure sensor, altitude sensor, gyro sensor, and lidar sensor built in the terminal 2. Note that the GPS data may include altitude data. Note that it is not limited to these types of sensors, and it can be similarly applied to other sensors.

[0041] The terminal 2 transmits the acquired image information and sensor information of the object to the server 1. In this embodiment, an example in which the terminal 2 transmits the image information of the object to the server 1 has been described, but the present invention is not limited thereto. For example, the terminal 2 may transmit an image of the object to the server 1. In this case, the server 1 acquires image information based on the image of the object transmitted from the terminal 2.

[0042] Based on the image information and sensor information of the object transmitted from the terminal 2, the server 1 refers to the image information and sensor information corresponding to a plurality of object candidates stored in the object DB 151, and identifies the object photographed from among the plurality of object candidates.

[0043] Specifically, the server 1 calculates the similarity between the object and each object candidate for each object candidate based on the image information of the object, the plurality of sensor information detected by each sensor incorporated in the terminal 2, the image information of each object candidate stored in the object DB 151, and the plurality of sensor information. For example, the server 1 calculates the similarity based on each evaluation item related to the image information and the plurality of sensor information of the object, and the weighting corresponding to each evaluation item.

[0044] FIG. 6 is an explanatory diagram for explaining the process of calculating the similarity based on the weighting. The server 1 acquires each evaluation item related to the image information and the plurality of sensor information of the object, and the weighting corresponding to each evaluation item. The evaluation items are provided based on the image information and the sensor information. As shown in the figure, the server 1 acquires evaluation items including "feature points", "character string", "wireless LAN information", "altitude", and "gyro".

[0045] Server 1 acquires the weighting corresponding to each evaluation item. The weighting is a coefficient representing the correlation with the importance of the evaluation item. For example, according to the importance, a five-level weighting is provided for each evaluation item, and the scores of the weighting at each level are 9 points, 7 points, 5 points, 3 points, and 1 point respectively. The weighting corresponding to each evaluation item may be stored in advance in the storage unit 12 or the mass storage unit 15, or the setting of the weighting may be accepted.

[0046] Regarding the acceptance process of the weighting setting, it may be performed on the server 1 side or the terminal 2 side. For example, the terminal 2 accepts the input of the weighting corresponding to each evaluation item and transmits the accepted weighting to the server 1. The server 1 receives the weighting transmitted from the terminal 2 and stores it in the storage unit 12 or the mass storage unit 15.

[0047] In addition, it is possible to accept the setting of the weighting corresponding to each evaluation item for each object. For example, for an object that is a switchboard, the five-level weighting for each evaluation item is "9 points, 7 points, 5 points, 3 points, and 1 point". For an object that is an environmental measurement device, the five-level weighting for each evaluation item is "10 points, 8 points, 6 points, 4 points, and 2 points".

[0048] Based on the acquired each evaluation item and the weighting corresponding to each evaluation item, Server 1 calculates the score of each evaluation item of the similarity using a predetermined calculation formula. The calculation formula is not particularly limited as long as it can calculate the score of each evaluation item.

[0049] As an example of the calculation formula for the score in the "feature point" evaluation item, it is represented by the following formula (1). "Feature point" score = coefficient × coincidence rate × weighting... (1) The coefficient is an arbitrary numerical value (for example, 255) provided for the convenience of calculation. The coincidence rate is, for example, the comparison result of the feature points obtained using OpenCV (Open Source Computer Vision Library).

[0050] As an example of the formula for calculating the score in the "string" evaluation item, it is represented by the following formula (2). "String" score = coefficient × number of identical strings × weighting... (2) The coefficient is the same as the coefficient in formula (1). The number of identical strings is the number of identical strings for the strings included in the image information.

[0051] As an example of the formula for calculating the score in the "Wireless LAN information" evaluation item, it is represented by the following formula (3). "Wireless LAN information" score = coefficient × number of access points × weighting... (3) The coefficient is the same as the coefficient in formula (1). The number of access points is the number of matching access points among the plurality of access points.

[0052] As an example of the formula for calculating the score in the "Altitude" evaluation item, it is represented by the following formula (4). "Altitude" score = coefficient × altitude score × weighting... (4) The coefficient is the same as the coefficient in formula (1). The altitude score may be provided based on the altitude difference. The altitude difference is the difference between the detected value of the altitude of the object and the detected value of the altitude of the object candidate. For example, when the altitude difference is less than 5m, it is determined as the same floor in the building, and the altitude score is 1 point. When the altitude difference is 5m or more, it is determined as different floors in the building, and the altitude score is 0 point.

[0053] As an example of the formula for calculating the score in the "Gyro" evaluation item, it is represented by the following formula (5). "Gyro" score = coefficient × angular velocity coefficient × weighting... (5) The coefficients are the same as those in formula (1). The angular velocity is the angle of rotation per unit time. The angular velocity coefficient is an index indicating the change in angular velocity. Using the rotation angles in the X-axis, Y-axis, and Z-axis directions, the angular velocity coefficient may be calculated by angular velocity coefficient = 1 - (X - Y - Z) / 360. For example, when the terminal 2 is not rotating in any of the three-axis directions, the angular velocity coefficient calculated by angular velocity coefficient = 1 - (0 - 0 - 0) / 360 is 1. Or, when the terminal 2 rotates 10° in the X-axis direction, 40° in the Y-axis direction, and does not rotate in the Z-axis direction, the angular velocity coefficient calculated by angular velocity coefficient = 1 - (10 - 40 - 0) / 360 is 0.92.

[0054] When the server 1 acquires the weightings corresponding to the evaluation items of "feature points", "character strings", "wireless LAN information", "altitude", and "gyro", as well as the image information and sensor information, it calculates the scores of each evaluation item using the above-described calculation formulas (1) to (5). Note that the server 1 may calculate the scores of at least one of the evaluation items of "feature points", "character strings", "wireless LAN information", "altitude", and "gyro". The server 1 calculates the total of the scores of each calculated evaluation item. The server 1 calculates the maximum value of the scores of each evaluation item. The maximum value of the score is calculated by multiplying the coefficient (for example, 255) by the weighting. For example, when the weighting corresponding to the "feature points" evaluation item is 9 points, the server 1 calculates that the maximum value of the score of this evaluation item is 2295 (255 × 9).

[0055] The server 1 calculates the similarity (total score / total maximum value of scores × 100%) between the object and the object candidate based on the total of the scores of each calculated evaluation item and the total of the maximum values of the scores of each evaluation item. As shown in the figure, the total score is 6024.8, and the total of the maximum values of the scores is 9435. The server 1 calculates that the similarity between the object and the object candidate is 63.86% (6024.8 / 9435 × 100%) based on the total score and the total of the maximum values of the scores.

[0056] In this way, the server 1 calculates the similarity between the object and each object candidate. The server 1 identifies the object candidate with the highest similarity as the photographed object. Alternatively, the object may be identified using a predetermined threshold value of the similarity. For example, when a predetermined threshold value of the similarity (e.g., 70%) is provided in advance, the server 1 determines whether the similarity between the object and the object candidate is equal to or greater than the predetermined threshold value. When the server 1 determines that the similarity is equal to or greater than the predetermined threshold value, the server 1 identifies the object candidate as the photographed object. When the server 1 determines that the similarity is less than the predetermined threshold value, the server 1 excludes the object candidate.

[0057] Note that the process is not limited to simply identifying the object candidate with the highest similarity as the photographed object. For example, the server 1 acquires object candidates up to a predetermined top rank (e.g., the third rank) in terms of similarity from a plurality of object candidates. In this way, a plurality of object candidates can be identified by ranking according to the similarity.

[0058] Note that the calculation process of the similarity is not limited to the process based on the weighting described above. For example, cosine similarity, Pearson's correlation coefficient, or deviation pattern similarity may be used as the similarity. In addition, the similarity can be calculated using a machine learning model. Note that the calculation process of the similarity by the machine learning model will be described in Embodiment 2.

[0059] The server 1 acquires information about the object from the object DB 151 based on the object ID of the identified object. The information about the object includes the object ID, the type of the object, or location information, etc. Note that the information about the object may include image information of the object and sensor information detected by the sensor of the terminal 2 when the object is photographed. The server 1 transmits the acquired information about the object to the terminal 2. The terminal 2 receives the information about the object transmitted from the server 1 and displays the received information about the object on the screen.

[0060] When the server 1 obtains object candidates up to a predetermined top rank in terms of similarity from a plurality of object candidates, the server 1 transmits information regarding all the obtained object candidates to the terminal 2. The terminal 2 receives the information regarding all the object candidates transmitted from the server 1 and displays the information regarding all the received object candidates on the screen. Note that since the information regarding the object candidates is the same as the information regarding the object, the description thereof is omitted.

[0061] FIG. 7 is an explanatory diagram showing an example of a screen for specifying an object. The specifying screen includes an image display column 11a, a shooting button 11b, an image selection button 11c, a specifying button 11d, a specifying result display column 11e, and a candidate display column 11f.

[0062] The image display column 11a is a display column for displaying an image of the object. The shooting button 11b is a button for shooting the object. The image selection button 11c is a button for selecting an image of the object. The specifying button 11d is a button for specifying the shot object from a plurality of object candidates. The specifying result display column 11e is a display column for displaying the specifying result of the object. The candidate display column 11f is a display column for displaying the specified object candidates ranked according to the similarity.

[0063] When the terminal 2 receives a touch operation on the shooting button 11b, the terminal 2 shoots the object via the shooting unit 26 and obtains an image of the shot object. When the terminal 2 receives a touch operation on the image selection button 11c, the terminal 2 obtains an image of the object stored in the storage unit 12 or the mass storage unit 15.

[0064] When the terminal 2 receives a touch operation on the specifying button 11d, the terminal 2 obtains image information (such as characters, feature points, or feature amounts) regarding the image based on the image of the object obtained by the shooting button 11b or the image selection button 11c. The terminal 2 obtains a plurality of sensor information (such as wireless LAN information, GPS data, barometric pressure data, altitude data, gyro sensor data, or lidar sensor data) detected by the sensors of the terminal 2 when the object is shot.

[0065] The terminal 2 transmits the acquired image information and a plurality of sensor information to the server 1. The server 1 receives the image information and a plurality of sensor information transmitted from the terminal 2. Based on the image information and a plurality of sensor information corresponding to the received object, and the image information and a plurality of sensor information corresponding to a plurality of object candidates stored in the object DB 151, the server 1 calculates the similarity between the object and each object candidate for each object candidate.

[0066] Based on the calculated similarity, the server 1 acquires object candidates up to a predetermined top rank (for example, the third rank) in terms of similarity from the plurality of object candidates. Based on the object ID of each acquired object candidate, the server 1 acquires information about each object candidate from the object DB 151. The information about the object candidate includes the object ID, the type of the object, owner information (for example, the name of the owner) of the object, or location information (for example, the location name or longitude and latitude), etc. The server 1 transmits the information about each acquired object candidate to the terminal 2.

[0067] The terminal 2 receives the information about each object candidate transmitted from the server 1 and displays it on the screen. Based on the similarity between the object and each object candidate for each object candidate, the terminal 2 displays the information about the object candidate with the highest similarity in the specific result display column 11e, and displays the information about the object candidate corresponding to the next highest similarity in the candidate display column 11f.

[0068] As shown in the figure, the object ID, the type of the object, the owner information (for example, Mr. A), and the location information (location name and longitude and latitude, etc.) of the object candidate with the highest similarity are displayed in the specific result display column 11e. The object ID, the type of the object, the owner information, and the location information of each object candidate corresponding to the next highest similarity are displayed in the candidate display column 11f. Note that the similarity of the object candidate may also be displayed in the specific result display column 11e and the candidate display column 11f.

[0069] In FIG. 7, an example in which three object candidates are acquired has been described, but the number of object candidates is not particularly limited. When the number of object candidates is large, for example, when a plurality of horizontally arranged object candidates cannot fit in the candidate display column 11f, the plurality of object candidates may be displayed so as to be scrollable horizontally.

[0070] FIG. 8 is a flowchart showing a processing procedure for specifying a photographed object. The control unit 21 of the terminal 2 photographs an object via the photographing unit 26 (step S201). When an image of the photographed object is stored in the storage unit 12 or the mass storage unit 15 in advance, the control unit 21 may acquire the image of the object from the storage unit 12 or the mass storage unit 15. The control unit 21 acquires image information (characters, feature points, feature amounts, etc.) regarding the image of the photographed object (step S202).

[0071] The control unit 21 acquires a plurality of sensor information (wireless LAN information, GPS data, barometric pressure data, altitude data, gyro sensor data, lidar sensor data, etc.) detected by the sensors of the terminal 2 at the time of photographing the object (step S203).

[0072] Specifically, the control unit 21 acquires wireless LAN information (MAC address of an access point, SSID, radio wave intensity, frequency, etc.) using a wireless communication method (for example, Wi-Fi Direct). The control unit 21 receives a transmission radio wave from a GPS positioning satellite via the GPS module 27 to acquire GPS data (latitude and longitude, etc.). The control unit 21 acquires barometric pressure data via the barometric pressure module 28. The control unit 21 acquires altitude data via the altitude module 29. The control unit 21 acquires gyro sensor data via the gyro module 30. The control unit 21 acquires lidar sensor data via the lidar module 31. Note that the present invention is not limited to these sensor types and can be similarly applied to other sensors.

[0073] The control unit 21 transmits the acquired image information of the object and the plurality of sensor information to the server 1 via the communication unit 23 (step S204). The control unit 11 of the server 1 receives the image information of the object and the plurality of sensor information transmitted from the terminal 2 via the communication unit 13 (step S101). The control unit 11 executes a subroutine of a process for calculating the similarity between the object and the object candidate based on the received image information and the plurality of sensor information (step S102). Note that the subroutine of the similarity calculation process will be described later.

[0074] Based on the calculated similarity, the control unit 11 identifies the object candidate with the highest similarity as the photographed object (step S103). Note that an object may be identified using a predetermined similarity threshold. For example, when the control unit 11 determines that the similarity between the object and the object candidate is equal to or higher than a predetermined threshold (for example, 70%), the control unit 11 identifies the object candidate as the photographed object.

[0075] Based on the object ID of the identified object, the control unit 11 acquires information about the object (such as the object ID, type, or position information) from the object DB 151 in the mass storage unit 15 (step S104). The control unit 11 transmits the acquired information about the object to the terminal 2 via the communication unit 13 (step S105). The control unit 21 of the terminal 2 receives the information about the object transmitted from the server 1 via the communication unit 23 (step S205). The control unit 21 displays the received information about the object on the display unit 25 (step S206) and ends the process.

[0076] FIG. 9 is a flowchart showing the processing procedure of the subroutine for calculating the similarity. The control unit 11 of the server 1 acquires each evaluation item related to the image information of the object and the plurality of sensor information from the storage unit 12 or the mass storage unit 15 (step S01). The evaluation items include, for example, "feature points", "character strings", "wireless LAN information", "altitude", and "gyro".

[0077] The control unit 11 receives the setting of the weighting corresponding to each evaluation item (step S02). Specifically, the control unit 21 of the terminal 2 receives the setting of the weighting corresponding to each evaluation item through the input unit 24, and transmits the received weighting to the server 1 through the communication unit 23. The control unit 11 of the server 1 receives the weighting corresponding to each evaluation item transmitted from the terminal 2 through the communication unit 13. Note that the control unit 11 of the server 1 may directly receive the setting of the weighting corresponding to each evaluation item.

[0078] In addition, when the weighting corresponding to each evaluation item is stored in advance in the storage unit 12 or the mass storage unit 15, the control unit 11 acquires the weighting corresponding to each evaluation item from the storage unit 12 or the mass storage unit 15. Also, it is possible to receive the setting of the weighting corresponding to each evaluation item for each object.

[0079] The control unit 11 acquires the image information and sensor information corresponding to a plurality of object candidates from the object DB 151 of the mass storage unit 15 (step S03). The control unit 11 acquires the image information and sensor information corresponding to one object candidate from the acquired plurality of object candidates (step S04). The control unit 11 calculates the score of each evaluation item using the above-described calculation formulas (1) to (5) based on the weighting corresponding to each evaluation item, the image information, and the plurality of sensor information (step S05).

[0080] The control unit 11 calculates the sum of the scores of each evaluation item calculated (step S06). The control unit 11 calculates the maximum value of the scores of each evaluation item by multiplying the coefficient (for example, 255) by the weighting (step S07). The control unit 11 calculates the similarity between the object and the object candidate (sum of scores / sum of maximum values of scores × 100%) based on the calculated sum of scores and the sum of the maximum values of scores (step S08).

[0081] The control unit 11 determines whether the target object candidate is the last target object candidate among a plurality of target object candidates (step S09). When the control unit 11 determines that the target object candidate is not the last target object candidate (NO in step S09), it returns to the process of step S04. When the control unit 11 determines that the target object candidate is the last target object candidate (YES in step S09), it acquires the similarity calculated for each target object candidate (step S10). The control unit 11 ends the subroutine of the similarity calculation process and returns.

[0082] According to the present embodiment, it is possible to identify the object captured from a plurality of target object candidates based on the image information and the sensor information.

[0083] According to the present embodiment, it is possible to output information regarding the identified object.

[0084] According to the present embodiment, it is possible to acquire target object candidates up to a predetermined top rank in terms of similarity from a plurality of target object candidates and output information regarding all the acquired target object candidates.

[0085] (Embodiment 2) Embodiment 2 relates to a form in which the similarity between an object and the target object candidate is output by artificial intelligence (AI: Artificial Intelligence) based on the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the target object candidate. Note that descriptions of the content overlapping with Embodiment 1 are omitted.

[0086] FIG. 10 is a block diagram showing a configuration example of the server 1 in Embodiment 2. Note that the same reference numerals are given to the content overlapping with FIG. 2 and the description thereof is omitted. The mass storage unit 15 of the server 1 includes a similarity output model 152 and a training data DB 153.

[0087] The similarity output model 152 is an output device that outputs (estimates) the similarity between an object and the object candidate based on the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the object candidate, and is a learned model generated by machine learning. The training data DB 153 stores training data for constructing (creating) the similarity output model 152.

[0088] FIG. 11 is an explanatory diagram showing an example of the record layout of the training data DB 153. The training data DB 153 includes an input data column and an output data column. The input data column includes an object column and an object candidate column. The object column stores the image information and sensor information corresponding to the object. The object candidate column stores the image information and sensor information corresponding to the object candidate. The output data column stores information on whether the object and the object candidate match (for example, match or do not match).

[0089] FIG. 12 is an explanatory diagram for explaining the similarity output model 152. The similarity output model 152 is used as a program module that is part of artificial intelligence software. The similarity output model 152 is an output device with a constructed neural network that takes as input the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the object candidate, and outputs the similarity between the object and the object candidate.

[0090] The neural network is, for example, a CNN (Convolutional Neural Network), and has an input layer that receives the input of the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the object candidate, an output layer that outputs the similarity between the object and the object candidate using an activation function, and an intermediate layer that has been learned by backpropagation. Each layer has one or more neurons (nodes), and each neuron has a value. And the neurons between a certain layer and the next layer are connected by edges, and each edge has variables (or parameters) such as weights and biases.

[0091] In a CNN, the values of neurons in each layer are obtained by performing a predetermined operation based on the values of neurons in the previous layer and the weights of edges, etc. When input data is input to the neurons in the input layer, the values of neurons in the next layer are obtained by a predetermined operation. Further, using the data obtained by the operation as input, the values of neurons in the next layer are obtained by the predetermined operation of that layer. Then, the values of neurons in the output layer, which is the final layer, become the output data for the input data.

[0092] For example, server 1 generates the similarity output model 152 using the training data stored in the training data DB153. Each record in the training data DB153 is training data. The values of the output data series are the correct data (information on whether the object and the object candidate match) to be output from the output layer. The image information and sensor information corresponding to the object in the input data series and the image information and sensor information corresponding to the object candidate are the input data.

[0093] The training data is data in the form of a combination in which the image information and sensor information corresponding to the object, the image information and sensor information corresponding to the object candidate, and the information on whether the object and the object candidate match are associated. The training data is generated based on a large amount of empirical data collected from the object. Note that the training data may be data created manually separately.

[0094] Server 1 performs learning using the acquired training data. Specifically, server 1 inputs the image information and sensor information corresponding to the object, which is the training data, and the image information and sensor information corresponding to the object candidate into the input layer, and through the arithmetic processing in the intermediate layer, outputs the similarity between the object and the object candidate from the output layer. The output layer includes, for example, a sigmoid function or a softmax function, and outputs the estimated similarity based on the feature amount output from the intermediate layer.

[0095] Server 1 compares the similarity (estimated result) output from the output layer with the correct value (value of the output data sequence) in the training data, and optimizes the parameters used in the arithmetic processing in the intermediate layer so that the output value from the output layer approaches the correct value. The parameters are, for example, the weights (coupling coefficients) between neurons, etc. The method for optimizing the parameters is not particularly limited. For example, Server 1 optimizes various parameters using the error backpropagation method.

[0096] Server 1 performs the above processing for each record stored in the training data DB153, and learns the similarity output model 152. Thereby, a model capable of outputting the similarity between the object and the object candidate can be constructed. Note that the above learning process may be performed by another computer (not shown) to generate the similarity output model 152. In this case, Server 1 acquires and installs the similarity output model 152 generated by another computer. Note that the similarity may be output by using a WEB API (Application Programming Interface) using a machine learning model without constructing the similarity output model 152.

[0097] When Server 1 acquires the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the object candidate, Server 1 inputs the acquired image information and sensor information corresponding to the object and the image information and sensor information corresponding to the object candidate into the similarity output model 152. Server 1 extracts the features of the input information by changing the number of dimensions of the input information input from the input layer in the intermediate layer of the similarity output model 152. Server 1 inputs the extracted features into the output layer of the similarity output model 152 and outputs an estimated result of estimating the similarity between the object and the object candidate.

[0098] As shown in the figure, for the image information and sensor information corresponding to the object, and the image information and sensor information corresponding to the object candidate, among the probability values of each similarity estimated by the similarity output model 152, the highest probability is output. For example, the estimated similarity is 75.4%. Note that it is not limited to the above-described output result, and the probability values of each estimated similarity may be directly output.

[0099] Note that the similarity output model 152 is not limited to CNN, and may be implemented by other models such as RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), SVM (Support Vector Machine), Bayesian network, Transformer network, regression tree, or random forest.

[0100] Figure 13 is a flowchart showing the processing procedure of a subroutine for calculating the similarity using the similarity output model 152. The control unit 11 of the server 1 acquires the image information and sensor information corresponding to a plurality of object candidates from the object DB 151 of the mass storage unit 15 (step S21). The control unit 11 acquires the image information and sensor information corresponding to one object candidate from the acquired plurality of object candidates (step S22).

[0101] The control unit 11 inputs the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the acquired object candidate into the similarity output model 152 (step S23). The control unit 11 outputs the similarity between the object and the object candidate from the similarity output model 152 (step S24).

[0102] The control unit 11 determines whether the target object candidate is the last target object candidate among a plurality of target object candidates (step S25). When the control unit 11 determines that the target object candidate is not the last target object candidate (NO in step S25), it returns to the process of step S22. When the control unit 11 determines that the target object candidate is the last target object candidate (YES in step S25), it acquires the similarity calculated for each target object candidate (step S26). The control unit 11 ends the subroutine of the similarity calculation process and returns.

[0103] According to the present embodiment, based on the image information and sensor information corresponding to the target object and the image information and sensor information corresponding to the target object candidate, it is possible to output the similarity between the target object and the target object candidate using the similarity output model 152.

[0104] According to the present embodiment, by outputting the similarity using the similarity output model 152, it is possible to obtain a highly accurate similarity.

[0105] The embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above meaning but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.

Explanation of Signs

[0106] 1 Information processing apparatus (server) 11 Control unit 12 Storage unit 13 Communication unit 14 Reading unit 15 Mass storage unit 151 Target object DB 152 Similarity output model 153 Training data DB 1a Portable storage medium 1b Semiconductor memory AP Control program 2 Information processing terminal (terminal) 21 Control unit 22 Memory unit 23 Communication unit 24 Input unit 25 Display unit 26 Imaging unit 27 GPS module 28 Barometric pressure module 29 Altitude module 30 Gyro module 31 LiDAR module 2P Control program

Claims

1. Acquire image information about the image of the object photographed by the information processing terminal; the information processing terminal is equipped with a plurality of sensors; acquiring a plurality of pieces of sensor information detected by each sensor of the information processing terminal when photographing the object; Obtaining each evaluation item related to the acquired image information and the plurality of sensor information, and a weighting corresponding to each evaluation item; calculating a similarity between the object and each of the object candidates stored in a storage unit that stores image information and sensor information corresponding to the plurality of object candidates, based on each of the acquired evaluation items and weights corresponding to each of the evaluation items; The candidate object with the highest similarity is identified as the photographed object. Information processing methods.

2. Accepts weighting settings corresponding to each of the evaluation items for each object. The information processing method according to claim 1 .

3. acquiring image information and sensor information corresponding to a plurality of candidate objects from the storage unit; inputting the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the acquired object candidate into a learning model that has been trained to output a similarity between the object and the object candidate when image information and sensor information corresponding to the object and image information and sensor information corresponding to the object candidate are input, and outputting a similarity for each object candidate; The candidate object with the highest similarity is identified as the photographed object. The information processing method according to claim 1 .

4. acquiring object candidates up to a predetermined top rank in terms of similarity from the plurality of object candidates; Print information about all captured object candidates 4. The information processing method according to claim 1.

5. Output information about the identified object 5. The information processing method according to claim 1.

6. The image information includes characters or feature points included in the image of the object, or feature amounts of the image.

6. The information processing method according to claim 1.

7. The sensor information includes wireless LAN information, satellite positioning system data, barometric pressure data, altitude data, gyro sensor data, or lidar sensor data.

7. The information processing method according to claim 1.

8. Acquire image information about the image of the object photographed by the information processing terminal; the information processing terminal is equipped with a plurality of sensors; acquiring a plurality of pieces of sensor information detected by each sensor of the information processing terminal when photographing the object; Obtaining each evaluation item related to the acquired image information and the plurality of sensor information, and a weighting corresponding to each evaluation item; calculating a similarity between the object and each of the object candidates stored in a storage unit that stores image information and sensor information corresponding to the plurality of object candidates, based on each of the acquired evaluation items and weights corresponding to each of the evaluation items; The candidate object with the highest similarity is identified as the photographed object. A program that causes a computer to perform a process.

9. An information processing device comprising a control unit, a plurality of sensors, and a memory unit, The control unit Acquire image information about the image of the object photographed by the information processing terminal; acquiring a plurality of pieces of sensor information detected by each sensor of the information processing terminal when photographing the object; Obtaining each evaluation item related to the acquired image information and the plurality of sensor information, and a weighting corresponding to each evaluation item; calculating a similarity between the object and each of the object candidates stored in the storage unit that stores image information and sensor information corresponding to the plurality of object candidates, based on each of the acquired evaluation items and weights corresponding to each of the evaluation items; The candidate object with the highest similarity is identified as the photographed object. Information processing device.

10. Acquiring image information relating to an image of an object photographed by an information processing terminal; acquiring sensor information detected by a sensor of the information processing terminal when photographing the object; acquiring image information and sensor information corresponding to the plurality of object candidates from a storage unit that stores image information and sensor information corresponding to the plurality of object candidates; inputting the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the acquired object candidate into a learning model that has been trained to output a similarity between the object and the object candidate when image information and sensor information corresponding to the object and image information and sensor information corresponding to the object candidate are input, and outputting a similarity for each object candidate; The candidate object with the highest similarity is identified as the photographed object. Information processing methods.

Images