Information processing method, information processing program, and information processing device.

Abstract

This invention provides an information processing method, an information processing device, and an information processing program that suppress the decrease in the detection accuracy of objects. [Solution] An information processing method performed on an information processing device equipped with at least one processor, comprising: generating movement data represented by the trajectory of an object from sensor data; correcting the parameters of the sensor data when the movement data represents the presence of an object; and detecting the object based on the corrected sensor data. The object is a person, robot, vehicle, various living organisms, a chair, a mobile terminal, and other objects, but is not limited to these.

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing method, an information processing program, and an information processing device.

Background Art

[0002] Conventionally, by analyzing sensor data such as a captured image, an object such as a person included in the sensor data has been detected. However, the detection accuracy of the object may decrease due to environmental changes in the sensing environment. A method of preparing a detection model for detecting an object from sensor data for each of a plurality of different types of sensing environments is also conceivable, but the development man-hours may increase. For this reason, it has been difficult to easily suppress a decrease in the detection accuracy of an object in the prior art.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention has been made in view of the above, and an object thereof is to provide an information processing method, an information processing program, and an information processing device that can easily suppress a decrease in the detection accuracy of an object.

Means for Solving the Problems

[0005] The information processing method according to the embodiment is an information processing method executed by an information processing device including at least one processor, generating movement line data represented by a trajectory of an object from sensor data, and when the movement line data represents a stay of the object, correcting parameters of the sensor data, and detecting the object based on the corrected sensor data.

Brief Description of the Drawings

[0006] [Figure 1] Figure 1 is a schematic diagram of an example of an information processing system according to an embodiment. [Figure 2A] Figure 2A is an explanatory diagram illustrating an example of parameter correction. [Figure 2B] Figure 2B is an explanatory diagram illustrating an example of parameter correction. [Figure 2C] Figure 2C is an explanatory diagram illustrating an example of a reference parameter. [Figure 3] Figure 3 is a flowchart showing an example of the information processing flow performed by the information processing device of the embodiment. [Figure 4] Figure 4 is a schematic diagram of an example of an information processing system based on a modified example. [Figure 5] Figure 5 is a hardware configuration diagram. [Modes for carrying out the invention]

[0007] The embodiments of the information processing method, information processing program, and information processing device will be described in detail below with reference to the attached drawings.

[0008] Figure 1 is a schematic diagram of an example of the information processing system 1 of this embodiment.

[0009] Information processing system 1 is a system for detecting object P.

[0010] Object P is the object to be detected in the information processing system 1. In this embodiment, Object P is an object from which movement data can be generated. Object P can be any object in which at least one of its shape and at least one of the positions of at least some of its constituent parts can change. This change may be due to the driving or movement of Object P itself, or it may be due to external forces such as being held or carried by another movable object. Object P is, for example, a person, a robot, a vehicle, various living things, an object such as a chair or a mobile terminal, etc., but is not limited to these.

[0011] In this embodiment, the form in which the object P is a person will be described as an example.

[0012] A movement path is a line that represents the trajectory of the positional change of an object P. The trajectory may be a line composed of a point cloud that is continuous in time series, or it may be a line composed of a point cloud that is discontinuous in at least part of it. A movement path may be a line that continues to be continuous in time series, or it may include a line, point, or point cloud that is discontinuously fragmented in at least part of it. Movement path data is data that represents a movement path.

[0013] The information processing system 1 comprises an information processing device 10 and a sensor. The information processing device 10 and the sensor are connected via a network NW or the like for communication.

[0014] A sensor is an element or device that collects information about an object P and outputs sensor data converted into a signal that can be handled by a machine. Sensor data represents physical quantities such as light, color, temperature, pressure, distance, speed, odor, and position.

[0015] The sensor can be any element or device that detects an object P included in the real space RS and collects information as sensor data that can generate movement data of the object P from the detection result. Specifically, the sensor may be an imaging device 20, an IR (Infrared) camera, an IR sensor, a distance sensor, etc., but is not limited to these.

[0016] In this embodiment, a configuration in which the sensor is an imaging device 20 will be described as an example. In this embodiment, a configuration in which the sensor data is captured image data will be described as an example.

[0017] The imaging device 20 acquires captured image data through imaging and transmits it to the information processing device 10. The imaging device 20 sequentially transmits the captured image data captured along the time series to the information processing device 10. That is, the imaging device 20 transmits captured video data composed of a plurality of captured image data to the information processing device 10. The captured image data is an example of sensor data. Hereinafter, the captured image data may be simply referred to as a captured image, and the captured video data may be simply referred to as a captured video for explanation. The captured image may also be referred to as a frame.

[0018] The imaging device 20 is arranged so as to be able to image a predetermined space in the real space. The predetermined real space is the real space RS where the object P to be detected can exist. The predetermined space is, for example, a space inside a building such as a factory, an outdoor space, a predetermined space where the environment such as brightness can vary, etc., but is not limited to these spaces.

[0019] In the present embodiment, an example will be described in which the imaging device 20 is fixedly arranged in the real space RS where the object P to be detected can exist. For this reason, the imaging device 20 has its angle of view, installation position, etc. adjusted in advance so as to be able to image the real space RS where the object P can exist. Also, the information processing system 1 may include a plurality of imaging devices 20. In FIG. 1, from the viewpoint of simplifying the explanation, a configuration in which the information processing system 1 includes one imaging device 20 is shown as an example. Also, the information processing system 1 is not limited to a form including only the imaging device 20 and may include a plurality of types of sensors.

[0020] The information processing device 10 is one or more information processing devices. The information processing device 10 is constituted by one or more dedicated or general-purpose computers.

[0021] The information processing apparatus 10 includes a communication unit 11, an input unit 12, an output device 13, a storage unit 14, and a processing unit 15. The communication unit 11, the input unit 12, the output device 13, the storage unit 14, and the processing unit 15 are communicably connected via a bus or the like. Also, the information processing apparatus 10 may be configured to include a photographing device 20. In this case, the photographing device 20 and the processing unit 15 may be communicably connected via a bus or the like.

[0022] The communication unit 11 communicates with the photographing device 20 and external information processing apparatuses via a network NW or the like. The input unit 12 receives various operations by the user. The input unit 12 is, for example, an input device such as a touch panel, a keyboard, a button, or the like. The output device 13 outputs various information. The output device 13 is a display that displays various information, a speaker that outputs various sounds, or the like. At least one of the input unit 12 and the output device 13 may be provided outside the information processing apparatus 10 and communicably connected to the processing unit 15.

[0023] The storage unit 14 stores various data. The storage unit 14 is, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, an optical disk, or the like. The storage unit 14 may be a storage device provided outside the information processing apparatus 10. Also, the storage unit 14 may be a storage medium that downloads and stores or temporarily stores a program and various information via a LAN (Local Area Network), the Internet, or the like.

[0024] The processing unit 15 executes information processing in the information processing apparatus 10.

[0025] The processing unit 15 includes an acquisition unit 15A, a correction unit 15B, a detection unit 15C, a flow line data generation unit 15D, an output control unit 15E, and an adjustment unit 15F.

[0026] The acquisition unit 15A, correction unit 15B, detection unit 15C, movement data generation unit 15D, output control unit 15E, and adjustment unit 15F are implemented by one or more processors. For example, each of the above units included in the processing unit 15 may be implemented by having a processor such as a CPU (Central processing unit) execute a program, i.e., by software. Alternatively, each of the above units included in the processing unit 15 may be implemented by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of these units may be implemented using a combination of software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0027] Alternatively, at least one of the above-mentioned components included in the processing unit 15 may be mounted on an external information processing device, such as a server device, which is connected to the information processing device 10 via a network NW or the like.

[0028] The acquisition unit 15A acquires sensor data. In this embodiment, the acquisition unit 15A acquires the captured image as sensor data.

[0029] In this embodiment, the imaging device 20 sequentially captures images at a predetermined frame rate in a time series and transmits the captured images sequentially to the information processing device 10 in the order they were captured.

[0030] The acquisition unit 15A acquires captured images taken by the imaging device 20. The acquisition unit 15A acquires captured video by sequentially receiving captured images from the imaging device 20. Alternatively, the acquisition unit 15A may acquire captured video by reading the captured video taken by the imaging device 20 and stored in the storage unit 14. The acquisition unit 15A sequentially stores the acquired sensor data, which are the captured images, in the storage unit 14.

[0031] The correction unit 15B corrects the parameters of the captured image, which is sensor data acquired by the acquisition unit 15A. In this embodiment, the correction unit 15B corrects the parameters of the captured image acquired by the acquisition unit 15A according to the parameter correction values ​​stored in the storage unit 14.

[0032] The correction unit 15B may perform parameter correction processing after adjusting the field of view, resolution, and other settings of the captured image acquired by the acquisition unit 15A.

[0033] A parameter is a physical quantity represented by sensor data. Parameters include elements that can be sensed by a sensor, such as brightness, contrast, hue, saturation, distance, and temperature. Brightness includes at least one of luminance and lightness.

[0034] As described above, in this embodiment, a configuration in which the sensor is the imaging device 20 will be described as an example. For this reason, a configuration in which the parameter is at least one of brightness, contrast, hue, and saturation will be described as an example.

[0035] A parameter correction value is a value used to correct the parameters of sensor data. In this embodiment, the parameter correction value is a value used to correct at least one of the parameters of a captured image, which is an example of sensor data: brightness, contrast, hue, and saturation. The parameter correction value is expressed as a value such as plus (+) N percent or minus (-) N percent, where N is an arbitrary number.

[0036] The memory unit 14 stores parameter correction values. More specifically, the memory unit 14 stores parameter correction values ​​for each type of parameter. Specifically, for example, the memory unit 14 stores parameter correction values ​​associated with each type of parameter, such as brightness, contrast, hue, and saturation.

[0037] The parameter correction values ​​are adjusted by the adjustment unit 15F, which will be described later (details below).

[0038] The correction unit 15B corrects the parameters of the captured image, which is sensor data acquired by the acquisition unit 15A, according to the parameter correction values ​​stored in the storage unit 14.

[0039] Figures 2A and 2B are explanatory diagrams illustrating an example of parameter correction for the captured image 30. The captured image 30 is an example of sensor data acquired by the acquisition unit 15A.

[0040] Figure 2A is a schematic diagram of an example of a captured image 30A acquired by the acquisition unit 15A. Captured image 30A is an example of captured image 30.

[0041] For example, let's assume that the captured image 30 acquired by the acquisition unit 15A is the captured image 30A shown in Figure 2A. Let's also assume that the parameter correction values ​​adjusted by the adjustment unit 15F, which will be described later, are stored in the storage unit 14 as parameter correction values ​​of "+40%" corresponding to the parameter "brightness" and parameter correction values ​​of "+40%" corresponding to the parameter "contrast".

[0042] In this case, the correction unit 15B corrects the value of the parameter "brightness" of the captured image 30A to a value of +40% according to the parameter correction value "+40%", and corrects the value of the parameter "contrast" of the captured image 30A to a value of +40% according to the parameter correction value "+40%".

[0043] The correction unit 15B corrects the parameters of the captured image 30A, generating a captured image 30B in which the brightness and contrast of the captured image 30A are corrected to +40% (see Figure 2B). Captured image 30B is an example of a captured image 30 whose parameters have been corrected by the correction unit 15B.

[0044] Returning to Figure 1, we continue the explanation.

[0045] The detection unit 15C detects the object P based on the captured image 30B (sensor data) corrected by the correction unit 15B.

[0046] The detection unit 15C detects the object P captured in the captured image 30B by analyzing the captured image 30B using a known image processing method. Even when the sensor data is not from the captured image 30, the detection unit 15C can detect the object P represented by the sensor data by analyzing the sensor data using a known method.

[0047] The detection unit 15C sequentially stores detection result information representing the detection result of the object P detected for each corrected captured image 30 in the storage unit 14 each time a corrected captured image 30B is generated.

[0048] The movement path data generation unit 15D generates movement path data, which is represented by the trajectory of the object P, from the sensor data.

[0049] Movement data is represented by the transition of the object P's two-dimensional position along the horizontal and depth directions. The horizontal direction is a direction along a horizontal plane perpendicular to the vertical direction. The depth direction is a direction along the horizontal plane and is perpendicular to the horizontal direction, which is a direction along the horizontal plane. Alternatively, movement data may be represented by the transition of the object P's three-dimensional position along the horizontal, depth, and height directions. The height direction is perpendicular to the horizontal direction. Furthermore, movement data may be represented by the transition of the object P's position along at least one of the horizontal, depth, and height directions.

[0050] In this embodiment, one example of a form in which the movement data is represented by the transition of the two-dimensional position of the object P along the horizontal and depth directions will be described.

[0051] The two-dimensional position of object P along the horizontal and depth directions, as represented by the movement data, is its position in real space RS. The two-dimensional position of object P along the horizontal and depth directions is represented, for example, by converting the position of the centroid of the detection frame of object P captured in the captured image 30 to a position in real space RS. A known conversion method can be used to convert the position in the captured image to a position in real space RS.

[0052] The movement path data generation unit 15D generates movement path data represented by the trajectory of the object P detected by the detection unit 15C. The movement path data generation unit 15D can generate the movement path data using a known method.

[0053] For example, when the motion data generation unit 15D detects an object P appearing in the captured image 30B, which is corrected sensor data, the detection unit 15C starts tracking the object P. The motion data generation unit 15D then converts the position of the center of gravity of the detection frame of the detected object P in the captured image 30B into a position in real space RS using a known method, thereby identifying the position of the object P in real space RS at the time the captured image 30B was taken. The detection frame is a frame that indicates that an object has been detected in the captured image 30B, and is a rectangular frame that surrounds the outline of the detected object P. The motion data generation unit 15D then tracks the object P for the captured image 30B, which has been corrected by sequentially acquiring parameters along the time series direction, and sequentially detects the center of gravity of each object P, thereby generating motion data that represents the change in the position of the center of gravity of the object P in real space RS.

[0054] The movement path data generation unit 15D may generate movement path data by analyzing the corrected video footage using a movement path extraction AI (Artificial Intelligence) that generates movement path data of the object P detected by the detection unit 15C from the captured video footage acquired by the acquisition unit 15A and corrected by the correction unit 15B. In other words, the movement path data generation unit 15D may generate movement path data by analyzing the corrected video footage using a movement path extraction AI. The movement path extraction AI is an AI that generates movement path data of the object P contained in the captured video footage. Any known AI can be used for the movement path extraction AI.

[0055] Furthermore, the movement path data generation unit 15D may generate movement path data from captured images 30 of the surrounding environment of the object P, which are captured by a shooting device 20 mounted on the object P, using SLAM (Simultaneous Localization And Mapping), and which have been corrected by the correction unit 15B. In this case, the shooting device 20 or a device such as a portable terminal equipped with the shooting device 20 may be carried or mounted on the object P. For SLAM, a known Visual-SLAM or the like may be used.

[0056] The output control unit 15E outputs the movement data generated by the movement data generation unit 15D to the output device 13. For example, the output control unit 15E displays an image on the output device 13 in which movement data representing the generated movement data is superimposed on the captured image 30 of the real space RS. Alternatively, the output control unit 15E may store the movement data generated by the movement data generation unit 15D in the storage unit 14.

[0057] The adjustment unit 15F adjusts the parameter correction value when the movement data generated by the movement data generation unit 15D represents the dwell time of the object P.

[0058] In detail, the adjustment unit 15F determines whether the movement data generated by the movement data generation unit 15D represents the stagnation of object P. Stagnation of object P means that object P remains stationary in approximately the same location even after a period of time has passed.

[0059] For example, the adjustment unit 15F determines that the movement data represents the object P lingering if the amount of movement of the object P per unit time, as represented by the movement data, is below a threshold.

[0060] This unit time is a predetermined number of frames, a predetermined time, etc. The predetermined time is, for example, 1 minute, but is not limited to this time. The predetermined number of frames is the number of frames captured by the imaging device 20 during the predetermined time, but is not limited to this number. This threshold only needs to be predetermined. The threshold only needs to be a value that allows it to be determined that the object P remains stationary in approximately the same location even after time has elapsed. Furthermore, the unit time, predetermined number of frames, predetermined time, and threshold can be changed as appropriate by user instructions for operation of the input unit 12, etc.

[0061] Furthermore, the adjustment unit 15F determines that if the movement data includes discontinuous periods, it indicates that the object P is lingering. More specifically, the adjustment unit 15F determines that if the movement data includes discontinuous periods because the object P is detected from a portion of multiple corrected captured images 30 (sensor data) in which at least a portion of the parameters are consecutive in different time series, it indicates that the object P is lingering. In other words, a discontinuous period is a period in which a portion of the object P's trajectory is discontinuous because the object P is detected from a portion of multiple corrected captured images 30 (sensor data) in which at least a portion of the parameters are consecutive in different time series, but the object P is not detected from a portion of the images. Specifically, a discontinuous period is a period in which the detection and non-detection of the object P are mixed, for example, when the object P is detected from a captured image 30 taken and corrected at a certain timing due to a change in brightness, but the object P is not detected from a captured image 30 taken at a different timing.

[0062] If the adjustment unit 15F determines that the movement data generated by the movement data generation unit 15D does not represent the dwell time of the object P, it does not adjust the parameter correction values ​​stored in the storage unit 14.

[0063] On the other hand, when the adjustment unit 15F determines that the movement data generated by the movement data generation unit 15D represents the dwell time of the object P, it adjusts the parameter correction value stored in the storage unit 14.

[0064] The adjustment unit 15F adjusts parameter correction values ​​to correct the parameters of the captured image 30, which is sensor data, so that they become parameters suitable for detection of the object P by the detection unit 15C.

[0065] For example, the adjustment unit 15F stores reference parameters in the storage unit 14 beforehand.

[0066] Reference parameters are parameters of the captured image 30 (sensor data) that enable the detection unit 15C to detect the object P with high accuracy.

[0067] Figure 2C is an explanatory diagram illustrating an example of a reference parameter.

[0068] Figure 2C shows the captured image 30C. Captured image 30C is an example of captured image 30. Captured image 30C is an example of sensor data sensed in a real-space RS environment suitable for detection of object P by the detection unit 15C. Figure 2C shows an example of captured image 30C taken in an environment in which object P included in real-space RS is clearly visible. The adjustment unit 15F stores the parameters of captured image 30C in the storage unit 14 as reference parameters. Specifically, for example, the adjustment unit 15F stores the value of the parameter "brightness" and the value of the parameter "contrast" of captured image 30C in the storage unit 14 as reference parameters, corresponding to the respective parameter types, "brightness" and "contrast".

[0069] Then, the adjustment unit 15F adjusts the parameter correction value based on the reference parameter when the movement data represents the dwell time of the object P.

[0070] Returning to Figure 1, the explanation continues. When the adjustment unit 15F determines that the movement data represents the presence of object P, it adjusts the parameter correction values ​​currently stored in the memory unit 14 to parameter correction values ​​that change the parameters of the newly acquired captured image 30 by a predetermined amount in a direction closer to the reference parameters.

[0071] For example, let's assume that the parameter correction value "+0%" corresponding to the parameter "brightness" and the parameter correction value "+0%" corresponding to the parameter "contrast" are currently stored in the memory unit 14. Let's also assume that the predetermined amount of change is, for example, "20%". Furthermore, let's assume that the parameters of the captured image 30, which is the latest sensor data acquired by the acquisition unit 15A, are less than the reference parameters.

[0072] In this case, the adjustment unit 15F adjusts the parameters of the captured image 30 to new parameter correction values: a parameter correction value of "+20%" corresponding to the parameter "brightness" and a parameter correction value of "+20%" corresponding to the parameter "contrast," as new parameter correction values ​​to change the parameters of the captured image 30 by a predetermined amount of change of "20%" in the direction closer to the reference parameters (in this case, the positive direction). Then, the adjustment unit 15F updates the parameter correction values ​​stored in the memory unit 14 with the newly adjusted parameter correction values. As a result, the parameter correction values ​​stored in the memory unit 14 are updated with the newly adjusted parameter correction values.

[0073] Furthermore, when the adjustment unit 15F determines that the movement data represents the presence of the object P, it adjusts the parameter correction values ​​currently stored in the memory unit 14 to parameter correction values ​​that correct the parameters of the newly acquired captured image 30 to match the reference parameters.

[0074] For example, let's assume that the parameter correction value "+0%" corresponding to the parameter "brightness" and the parameter correction value "+0%" corresponding to the parameter "contrast" are currently stored in the memory unit 14. Let's also assume that the parameters of the captured image 30, which is the latest sensor data acquired by the acquisition unit 15A, are in a state where correcting the parameter values ​​to +40% will result in values ​​that match the reference parameter values.

[0075] In this case, the adjustment unit 15F adjusts the parameter correction values ​​to change the parameters of the captured image 30 to values ​​that match the reference parameters. Specifically, it sets the parameter correction value "+40%" corresponding to the parameter "brightness" and the parameter correction value "+40%" corresponding to the parameter "contrast" as new parameter correction values. The adjustment unit 15F then updates the parameter correction values ​​stored in the memory unit 14 with the newly adjusted parameter correction values. As a result, the parameter correction values ​​stored in the memory unit 14 are updated with the newly adjusted parameter correction values.

[0076] Through these processes, if the movement data represents the dwell time of the object P, the parameter correction values ​​stored in the memory unit 14 are updated to parameter correction values ​​that correct the parameters of the captured image 30 to approach or match the reference parameters.

[0077] As described above, the correction unit 15B corrects the parameters of the captured image 30, which is sensor data acquired by the acquisition unit 15A, according to the parameter correction values ​​stored in the storage unit 14. Therefore, when the movement data represents the dwell time of the object P, the correction unit 15B can correct the parameters of the sensor data (captured image 30) to move closer to or match the reference parameters.

[0078] For example, consider a case where the acquisition unit 15A acquires the captured image 30A shown in Figure 2A, and the correction unit 15B corrects the captured image 30A according to the parameter correction values ​​stored in the storage unit 14. In this case, the correction unit 15B can generate a captured image 30B (see Figure 2B) in which the parameters of the captured image 30A shown in Figure 2A are corrected to match or approach the reference parameters of the captured image 30C shown in Figure 2C.

[0079] Returning to Figure 1, let's continue the explanation. As described above, the detection unit 15C detects the object P based on the captured image 30B (sensor data) corrected by the correction unit 15B.

[0080] As described above, the parameter correction value is a correction value used to adjust the parameters of the captured image 30 (sensor data) to match or approach the reference parameter. The reference parameter is the parameter at which the detection unit 15C can detect the object P from the captured image 30 (sensor data) with high accuracy.

[0081] Therefore, the detection unit 15C detects the object P based on the captured image 30B (sensor data) corrected by the correction unit 15B, making it easy to suppress a decrease in the detection accuracy of the object P.

[0082] Next, an example of the information processing flow performed by the information processing device 10 of this embodiment will be described.

[0083] Figure 3 is a flowchart showing an example of the information processing flow performed by the information processing device 10 of this embodiment.

[0084] The acquisition unit 15A acquires the captured images 30 (step S100). In this embodiment, the acquisition unit 15A sequentially acquires the captured images 30 as sensor data by sequentially acquiring the captured images 30 that are sequentially taken in chronological order by the imaging device 20.

[0085] The correction unit 15B reads the parameter correction value stored in the storage unit 14 each time the acquisition unit 15A acquires a new captured image 30 (step S102).

[0086] The correction unit 15B corrects the parameters of the captured image 30 acquired in step S100 according to the parameter correction value read in step S102 (step S104).

[0087] In step S104, for example, the parameters of the captured image 30A (see Figure 2A) acquired by the acquisition unit 15A are corrected to match or be closer to the reference parameters of the captured image 30C (see Figure 2C), which is sensor data sensed in a real-space RS environment suitable for detection of the object P by the detection unit 15C, and a captured image 30B (see Figure 2B) is generated.

[0088] The detection unit 15C detects the object P that appears in the captured image 30B (sensor data) whose parameters were corrected in step S104 (step S106).

[0089] The movement data generation unit 15D generates movement data represented by the trajectory of the object P detected in step S106 (step S108).

[0090] The output control unit 15E outputs the movement path data generated in step S108 (step S110). For example, the output control unit 15E displays an image on the output device 13 in which movement paths representing the movement path data generated in step S108 are superimposed on the captured image 30 of the real space RS. Alternatively, the output control unit 15E may store the movement path data generated by the movement path data generation unit 15D in the storage unit 14.

[0091] The adjustment unit 15F determines whether the movement data generated in step S108 represents the presence of object P (step S112). If the determination in step S112 is negative, i.e., if the movement data does not represent the presence of object P (step S112: No), the process proceeds to step S116. If the determination in step S112 is positive, i.e., if the movement data represents the presence of object P (step S112: Yes), the process proceeds to step S114.

[0092] In step S114, the adjustment unit 15F adjusts the parameter correction value based on the reference parameter (step S114). Then, the adjustment unit 15F updates the parameter correction value currently stored in the memory unit 14 to the adjusted parameter correction value.

[0093] The processing unit 15 determines whether or not to terminate the process (step S116). For example, the processing unit 15 makes the determination in step S116 by determining whether or not a termination instruction has been input by the user through an operation instruction on the input unit 12. If the determination in step S116 is negative (step S116: No), the process returns to step S100. If the determination in step S116 is positive (step S116: Yes), the routine terminates.

[0094] As described above, the information processing method of this embodiment is an information processing method executed by an information processing device 10 equipped with at least one processor, which generates movement data represented by the trajectory of an object P from sensor data (captured image 30), corrects the parameters of the sensor data (captured image 30) when the movement data represents the presence of the object P, and detects the object P based on the corrected sensor data (captured image 30).

[0095] In conventional technology, the detection accuracy of object P could decrease due to environmental fluctuations in the sensing environment. While it is conceivable to pre-prepare detection models for multiple different types of sensing environments to detect object P from sensor data, this could increase development time. Therefore, it was difficult to easily suppress the decrease in detection accuracy of object P using conventional technology.

[0096] On the other hand, in the information processing method executed by the information processing device 10 of this embodiment, if the movement data generated based on the object P detected from the sensor data represents the presence of the object P, the parameters of the sensor data are corrected. Then, in the information processing method of this embodiment, the object P is detected based on the corrected sensor data.

[0097] Therefore, in the information processing method performed by the information processing device 10 of this embodiment, by using movement data, it is possible to easily suppress a decrease in the detection accuracy of the target object P with a simple configuration.

[0098] Therefore, the information processing method performed by the information processing device 10 of this embodiment can easily suppress a decrease in the detection accuracy of the target object P.

[0099] Furthermore, in the information processing method executed by the information processing device 10 of this embodiment, if the amount of movement of the object P per unit time represented by the movement data is below a threshold, it is determined that the movement data represents the object P lingering.

[0100] Therefore, in the information processing method of this embodiment, even though an object is actually moving in real space RS, it does not appear clearly as a physical quantity in the sensor data. As a result, when the amount of movement of the object P represented by the movement path data is below a threshold, it becomes easy to determine that parameter correction is necessary for high-precision detection of the object P.

[0101] Furthermore, in the information processing method of this embodiment, if the movement data includes discontinuous periods, it is determined that this represents the presence of object P. In addition, in the information processing method of this embodiment, if object P is detected from a portion of multiple corrected sensor data that are consecutive in different time series of parameters, and the movement data includes discontinuous periods, it is determined that this represents the presence of object P.

[0102] Therefore, in the information processing method of this embodiment, even though the object P is actually continuously located within the range that can be sensed by the sensor (imaging device 20) in real space RS, it is possible to easily determine that a state in which the object P appears or does not appear as a physical quantity in the sensor data due to fluctuations in the sensing environment is a state in which parameter correction is necessary for high-precision detection of the object P.

[0103] Furthermore, in the information processing method of this embodiment, the parameters of the sensor data (captured image 30) are corrected according to the parameter correction value, the object P is detected based on the corrected sensor data, and movement data represented by the trajectory of the detected object P is generated. Then, in the information processing method of this embodiment, if the movement data represents the dwelling of the object P, the parameter correction value is adjusted.

[0104] Therefore, in the information processing method of this embodiment, sensor data acquired sequentially in time series is corrected according to parameter correction values ​​adjusted based on sensor data acquired previously, and the target object P can be detected from the corrected sensor data.

[0105] Therefore, the information processing method of this embodiment can easily suppress a decrease in the detection accuracy of the target object P.

[0106] Furthermore, in the information processing method of this embodiment, if the movement data represents the dwell time of the object P, the parameter correction value is adjusted based on the reference parameter. In addition, in the information processing method of this embodiment, the parameter correction value is adjusted so that the parameters of the sensor data (captured image 30) become the reference parameter.

[0107] Therefore, in the information processing method of this embodiment, the detection unit 15C pre-sets parameters that enable high-precision detection of the target object P from the sensor data (captured image 30) as reference parameters, thereby enabling even higher-precision detection of the target object P in addition to the above-mentioned effects.

[0108] (Variation 1) In the above embodiment, as shown in Figure 1, an example configuration was described in which the information processing system 1 comprises an information processing device 10 and a camera 20. However, at least one of the above components included in the processing unit 15 of the information processing device 10 may be mounted on an external information processing device such as a server device that is communicably connected to the information processing device 10 via a network NW or the like. Alternatively, the camera 20 may be mounted on an external information processing device that is communicably connected to the information processing device 10.

[0109] Figure 4 is a schematic diagram of an example of the information processing system 1B of this modified example.

[0110] In this modified example, the same reference numerals are used for components identical to those in the above embodiment, and detailed descriptions are omitted.

[0111] The information processing system 1B comprises an information processing device 10B, a terminal device 21, and a server device 40. The information processing device 10B, the terminal device 21, and the server device 40 are connected to each other via a network NW or the like.

[0112] The terminal device 21 is one or more information processing devices. The terminal device 21 is composed of one or more dedicated or general-purpose computers. The terminal device 21 includes a camera 20. The camera 20 is the same as in the above embodiment.

[0113] The terminal device 21 sequentially transmits the captured images 30, which were captured in chronological order by the imaging device 20, to the information processing device 10B.

[0114] The information processing device 10B is one or more information processing devices. The information processing device 10 is composed of one or more dedicated or general-purpose computers.

[0115] The information processing device 10B is the same as the information processing device 10 of the above embodiment, except that it includes a processing unit 16 instead of a processing unit 15.

[0116] The processing unit 16 performs information processing in the information processing device 10B. The processing unit 16 includes an acquisition unit 15A, a correction unit 15B, a detection unit 15C, an output control unit 16G, and an adjustment unit 15F. The processing unit 16 is the same as the processing unit 15 in the above embodiment, except that it does not include a movement data generation unit 15D and has an output control unit 16G instead of an output control unit 15E.

[0117] The output control unit 16G sequentially stores detection result information representing the detection result of the object P detected for each corrected captured image 30 by the detection unit 15C in the storage unit 14 and transmits it to the server device 40. The output control unit 16G also transmits the captured image 30 used to detect the object P to the server device 40.

[0118] Then, each time the output control unit 16G transmits the detection result information and the captured image 30 to the server device 40, it receives the movement path data generated by the server device 40 based on the detection result information and the captured image 30.

[0119] The output control unit 16G then outputs the received movement data to the output device 13, similar to the output control unit 15E. For example, the output control unit 16G displays an image on the output device 13 in which movement data representing the movement data generated is superimposed on the captured image 30 of the real space RS. Alternatively, the output control unit 15E may store the received movement data in the storage unit 14.

[0120] The adjustment unit 15F is the same as in the above embodiment.

[0121] The server device 40 is one or more information processing devices. The server device 40 is composed of one or more dedicated or general-purpose computers.

[0122] The server device 40 comprises a communication unit 41, an input unit 42, an output device 43, a storage unit 44, and a processing unit 45. The communication unit 41, the input unit 42, the output device 43, the storage unit 44, and the processing unit 45 are communicated to each other via a bus or the like.

[0123] The communication unit 41 communicates with the information processing device 10B and other external information processing devices via a network NW or the like. The input unit 42 accepts various operations from the user. The input unit 42 is, for example, an input device such as a touch panel, keyboard, or buttons. The output device 43 outputs various types of information. The output device 43 is a display that shows various types of information, a speaker that outputs various types of sound, etc. At least one of the input unit 42 and the output device 43 may be configured to be located outside the server device 40 and connected to the processing unit 45 in a manner that allows communication.

[0124] The storage unit 44 stores various types of data. The storage unit 44 may be, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, or an optical disc. The storage unit 44 may also be a storage device located outside the server device 40. Alternatively, the storage unit 44 may be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN or the Internet.

[0125] The processing unit 45 performs information processing on the server device 40.

[0126] The processing unit 45 comprises an acquisition unit 45A, a movement path data generation unit 45B, and an output control unit 45C. The acquisition unit 45A, the movement path data generation unit 45B, and the output control unit 45C are implemented by one or more processors. For example, each of the above units included in the processing unit 45 may be implemented by having a processor such as a CPU execute a program, i.e., by software. Alternatively, each of the above units included in the processing unit 45 may be implemented by a dedicated IC or other processor, i.e., by hardware. Each of these units may be implemented using a combination of software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0127] The acquisition unit 45A sequentially acquires detection result information representing the detection result of the object P, and the captured image 30 used to detect the object P, from the information processing device 10B.

[0128] The movement path data generation unit 45B generates movement path data represented by the trajectory of the object P detected by the detection unit 15C. The movement path data generation unit 45B can generate movement path data in the same manner as the movement path data generation unit 15D, using the detection result information and captured image 30 received from the information processing device 10B.

[0129] The output control unit 45C transmits the movement path data generated by the movement path data generation unit 45B to the information processing device 10B. The output control unit 45C may also display an image on the output device 43 in which the movement path representing the movement path data generated by the movement path data generation unit 45B is superimposed on the captured image 30 of the real space RS. The output control unit 45C may also store the generated movement path data in the storage unit 44.

[0130] Thus, in this modified information processing system 1B, the generation of movement data is performed by a server device 40 external to the information processing device 10B.

[0131] In this modified information processing system 1B, sensor data such as captured images 30 are stored within the information processing device 10B and not transmitted externally. Instead, the generation process of movement data can be performed on a server device 40 outside the information processing device 10B. Therefore, in addition to the effects of the above embodiment, this modified information processing system 1B can reduce the processing load on the information processing device 10B while maintaining the confidentiality of sensor data.

[0132] Next, an example of the hardware configuration of the information processing device 10 of the above embodiment and the information processing device 10B of the modified example will be described.

[0133] Figure 5 is a hardware configuration diagram of an example of the information processing device 10 of the above embodiment and the information processing device 10B of the modified example described above.

[0134] The information processing device 10 of the above embodiment and the information processing device 10B of the above modified example have a CPU (Central Processing Unit) 80, ROM (Read Only Memory) 82, RAM (Random Access Memory) 84, and I / F 86 etc. interconnected by a bus 88, and have a hardware configuration that uses a normal computer.

[0135] The CPU 80 is an arithmetic unit that controls the information processing device 10 of the above embodiment and the modified information processing device 10B described above. The ROM 82 stores programs and the like that realize information processing by the CPU 80. The RAM 84 stores data necessary for various processes performed by the CPU 80. The I / F 86 is an interface connected to the storage unit, input unit, output unit, sensor, and communication unit, etc., for sending and receiving data.

[0136] In the information processing device 10 of the above embodiment and the modified information processing device 10B described above, the CPU 80 reads a program from the ROM 82 onto the RAM 84 and executes it, thereby realizing each of the above-mentioned functional units on the computer.

[0137] The programs for executing the above-mentioned processes performed by the information processing device 10 of the above embodiment and the information processing device 10B of the modified example may be stored in an HDD (hard disk drive). Alternatively, the programs for executing the above-mentioned processes performed by the information processing device 10 of the above embodiment and the information processing device 10B of the modified example may be pre-installed and provided in a ROM 82.

[0138] Furthermore, the program for executing the above processing performed by the information processing device 10 of the above embodiment and the information processing device 10B of the modified example may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD), and provided as a computer program product. Alternatively, the program for executing the above information processing performed by the information processing device 10 of the above embodiment and the information processing device 10B of the modified example may be stored on a computer connected to a network such as the Internet and provided by allowing download via the network. Alternatively, the program for executing the above information processing performed by the information processing device 10 of the above embodiment and the information processing device 10B of the modified example may be provided or distributed via a network such as the Internet.

[0139] Although embodiments and modifications have been described above, these embodiments and modifications are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments and modifications can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0140] Furthermore, this technology can also be configured as follows. (1) An information processing method performed on an information processing device having at least one processor, From sensor data, motion data is generated, which is represented by the trajectory of the object. If the aforementioned movement data represents the dwelling of the object, the parameters of the sensor data are corrected. The object is detected based on the corrected sensor data. Information processing methods. (2) If the amount of movement of the object per unit time represented by the movement data is below a threshold, it is determined that the movement data represents the object lingering. (1) The information processing method described above. (3) If the aforementioned movement data includes a discontinuous period, it is determined that this represents the dwelling of the object. The information processing method described in (1) or (2). (4) If at least some of the parameters are detected from a portion of a plurality of corrected sensor data that are consecutive in different time series, and the movement data includes discontinuous periods, it is determined that this represents the presence of the object. (3) The information processing method described above. (5) The parameters of the aforementioned sensor data are corrected according to the parameter correction value. Based on the corrected sensor data, the object is detected. The movement data is generated, which is represented by the trajectory of the detected object. If the movement data represents the dwelling of the object, adjust the parameter correction value. An information processing method described in any one of (1) to (4). (6) If the aforementioned movement data represents the dwelling of the object, the parameter correction value is adjusted based on the reference parameter. (5) The information processing method described above. (7) The parameter correction value is adjusted to correct the sensor data so that the parameter becomes a reference parameter. The information processing method described in (5) or (6). (8) The parameter is at least one of brightness, contrast, hue, and saturation. An information processing method described in any one of (1) to (7). (9) A movement data generation unit generates movement data represented by the trajectory of an object from sensor data, When the movement data indicates the presence of the object, a correction unit corrects the parameters of the sensor data, A detection unit that detects the object based on the corrected sensor data, An information processing device equipped with the following features. (10) A step of generating movement data represented by the trajectory of an object from sensor data, If the movement data indicates the presence of the object, the steps include correcting the parameters of the sensor data, A step of detecting the object based on the corrected sensor data, An information processing program that causes a computer to execute something. [Explanation of Symbols]

[0141] 10, 10B Information Processing Device 15, 16, 45 Processing Unit 15B Correction section 15C detection unit 15D, 45B Movement path data generation unit 15F Adjustment section

Claims

1. An information processing method performed on an information processing device having at least one processor, From sensor data, motion data is generated, which is represented by the trajectory of the object. If the aforementioned movement data represents the dwelling of the object, the parameters of the sensor data are corrected. The object is detected based on the corrected sensor data. Information processing methods.

2. If the amount of movement of the object per unit time represented by the movement data is below a threshold, it is determined that the movement data represents the object lingering. The information processing method according to claim 1.

3. If the aforementioned movement data includes a discontinuous period, it is determined that this represents the dwelling of the object. The information processing method according to claim 1.

4. If at least some of the parameters are detected from a portion of a plurality of corrected sensor data that are consecutive in different time series, and the movement data includes discontinuous periods, it is determined that this represents the presence of the object. The information processing method according to claim 3.

5. The parameters of the aforementioned sensor data are corrected according to the parameter correction value. Based on the corrected sensor data, the object is detected. The movement data is generated, which is represented by the trajectory of the detected object. If the movement data represents the dwelling of the object, adjust the parameter correction value. The information processing method according to claim 1.

6. If the aforementioned movement data represents the dwelling of the object, the parameter correction value is adjusted based on the reference parameter. The information processing method according to claim 5.

7. The parameter correction value is adjusted to correct the sensor data so that the parameter becomes a reference parameter. The information processing method according to claim 5.

8. The parameter is at least one of brightness, contrast, hue, and saturation. The information processing method according to claim 1.

9. A movement data generation unit generates movement data represented by the trajectory of an object from sensor data, When the movement data indicates the presence of the object, a correction unit corrects the parameters of the sensor data, A detection unit that detects the object based on the corrected sensor data, An information processing device equipped with the following features.

10. A step of generating movement data represented by the trajectory of an object from sensor data, If the movement data indicates the presence of the object, the steps include correcting the parameters of the sensor data, A step of detecting the object based on the corrected sensor data, An information processing program that causes a computer to execute something.

Images