Information processing device, information processing method, program and system

Abstract

To provide an information processor, an information processing method, a program, and a system that can appropriately recognize the distance to an article and the shape of the article.SOLUTION: An information processor includes a sensor interface and a processor. The sensor interface connects with an ultrasonic transmission element that irradiates an article with an ultrasonic wave and an ultrasonic reception element that receives a reflected wave from the article. The processor emits an ultrasonic wave having a first frequency and an ultrasonic wave having a second frequency different from the first frequency from the ultrasonic transmission element through the sensor interface, acquires intensities of a first reflected wave of the ultrasonic wave having the first frequency and a second reflected wave of the ultrasonic wave having the second frequency received by the ultrasonic reception element, and generates shape information indicating a shape of the article on the basis of the acquired intensities of the first reflected wave and the second reflected wave.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus, an information processing method, a program, and a system.

Background Art

[0002] A system for moving articles using robots is provided in a logistics center or the like. Such a system recognizes an article (for example, the type, shape, or position of the article) based on an image of the article and picks up the article.

[0003] Conventionally, when an article is a black box or a transparent box, or is wrapped with a transparent film such as a wrapping sheet, the system may not be able to appropriately recognize the distance to the article and the shape of the article.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to solve the above problems, there is provided an information processing apparatus, an information processing method, a program, and a system capable of appropriately recognizing the distance to an article and the shape of the article.

Means for Solving the Problems

[0006] According to one embodiment, the information processing device comprises a sensor interface and a processor. The sensor interface is connected to an ultrasonic transmitting element that irradiates an article with ultrasonic waves and an ultrasonic receiving element that receives reflected waves from the article. The processor irradiates the article with ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency via the sensor interface, obtains the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by the ultrasonic receiving element, and generates shape information indicating the shape of the article based on the obtained intensity of the first reflected wave and the intensity of the second reflected wave. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows an example of the configuration of a cargo handling system according to the first embodiment. [Figure 2] Figure 2 shows an example of the configuration of a sensor device according to the first embodiment. [Figure 3] Figure 3 is a block diagram showing an example of the configuration of the control system of a cargo handling system according to the first embodiment. [Figure 4] Figure 4 shows an example of the operation of the sensor device according to the first embodiment. [Figure 5] Figure 5 is a graph showing the relationship between received signal strength and elapsed time according to the first embodiment. [Figure 6] Figure 6 shows an example of operation of the sensor device according to the first embodiment. [Figure 7] Figure 7 is a graph showing the relationship between the received signal strength and the ultrasonic frequency according to the first embodiment. [Figure 8] Figure 8 shows the relationship between received signal strength and shape according to the first embodiment. [Figure 9] Figure 9 shows an example of shape information according to the first embodiment. [Figure 10] Figure 10 is a flowchart showing an example of the operation of the cargo handling system according to the first embodiment. [Figure 11]Figure 11 shows an example of the configuration of a sensor device according to the second embodiment. [Modes for carrying out the invention]

[0008] The embodiments will be described below with reference to the drawings. (First embodiment) First, let me describe the first embodiment. The cargo handling system according to this embodiment moves goods using cargo handling equipment such as a robot. The cargo handling system recognizes goods placed in a predetermined area. The cargo handling system uses a robot to grasp the recognized goods and load them onto a conveyor belt or the like. For example, the cargo handling system is used in a logistics center or warehouse.

[0009] Figure 1 schematically shows an example of the configuration of a cargo handling system 1 according to an embodiment. As shown in Figure 1, the cargo handling system 1 comprises a cargo handling device 10, a sensor device 20, a camera 30, and a frame 40. The cargo handling device 10 also comprises a moving device 50 and a control device 60. The control device 60, camera 30, sensor device 20, and moving device 50 are connected, for example, via a serial cable or LAN (Local Area Network). In the following description, the vertical direction refers to the vertical direction of the cargo handling system 1.

[0010] The frame 40 has a rectangular lower frame member 41, a rectangular upper frame member 42, and four (or more) column members 43. The lower frame member 41 is fixed to the floor surface G. The upper frame member 42 is provided above the lower frame member 41. The four column members 43 connect the four corners of the lower frame member 41 and the four corners of the upper frame member 42. The pallet 2 is placed on the floor surface G within the lower frame member 41. That is, the pallet 2 is placed inside the frame 40. The floor surface G extends horizontally.

[0011] The pallet 2 is positioned at a prescribed position with reference to the frame 40. The area of the upper surface of the pallet 2 positioned at the prescribed position is the placement area R1 of the article W1. Note that the placement area R1 is not limited to this. The placement area R1 may be provided, for example, on the floor surface G or the like. A plurality of articles W1 can be placed on the placement area R1. The article W1 is, for example, a PET bottle, a box, or the like. In the present embodiment, when referring to a plurality of packages including one or more articles W1, it is called an article group W. The pallet 2 may be moved by a forklift or the like.

[0012] The handling device 10 is a device for gripping the article W1 placed on the placement area R1 of the pallet 2 and moving it to a predetermined position (destination). For example, the handling device 10 loads the article W1 onto a conveyor or the like as the destination.

[0013] As described above, the handling device 10 includes a moving device 50 and a control device 60. The moving device 50 is arranged around the pallet 2 and the destination.

[0014] The moving device 50 moves the article W1 according to the control of the control device 60. The moving device 50 includes a base 111, an arm portion 112 supported by the base 111, and a gripping portion 113 provided at the tip of the arm portion 112.

[0015] The base 111 is a housing fixed to the floor surface G. The arm portion 112 is, as an example, a multi-joint robot arm. The arm portion 112 is configured to be able to execute an operation of moving the article W1 held by the gripping portion 113 in the vertical direction and an operation of moving it in the horizontal direction.

[0016] For example, the arm portion 112 is composed of three movable parts connected by a plurality of joints that can rotate around one or two axes.

[0017] A gripping part 113 is attached to the tip of the arm part 112. The gripping part 113 sandwiches and grips one article W1. The gripping part 113 may include a pair of claw parts. Note that the gripping of the article W1 is not limited to the gripping method by sandwiching. For example, the gripping part 113 may adsorb and grip the article W1 by negative pressure. Also, the gripping part 113 may grip two articles W1 at a time.

[0018] The control device 60 (information processing device) is a device that controls the sensor device 20, the camera 30, and the moving device 50. That is, the control device 60 is a device that controls the entire material handling system 1. The control device 60 controls the operation of the moving device 50 based on the imaging result of the camera 30 and the sensing result of the sensor device 20. The control device 60 may be provided integrally with the moving device 50 or may be provided separately from the moving device 50. The control device 60 will be described in detail later.

[0019] In this embodiment, the moving device 50 and the control device 60 are used as the material handling devices, but not limited thereto, the control device 60 may be used as the material handling device.

[0020] The camera 30 is provided above the pallet 2. Specifically, the camera 30 is fixed to the upper frame member 42 of the frame 40 via an attachment member or the like. In the example shown in FIG. 1, the camera 30 is disposed at the center or the end within the frame of the upper frame member 42. The camera 30 is, as an example, a 3D camera. For example, the camera 30 is a distance sensor that optically measures the distance to the target part by a stereo camera or a laser or the like. The camera 30 is an example of an imaging device.

[0021] The camera 30 images the article group W placed in the placement area R1 of the pallet 2 from above. The camera 30 generates optical distance information indicating the distance at each point. The camera 30 outputs the generated optical distance information to the control device 60. The imaging range of the camera 30 is set to include at least the article group W.

[0022] The sensor device 20 irradiates the group of items W with ultrasonic waves and receives the reflected ultrasonic waves (reflected waves). The sensor device 20 irradiates the group of items W from above the pallet 2, including the area containing the group of items W. The sensor device 20 outputs a sensing result indicating the intensity of the reflected waves to the control device 60. The imaging range of the camera 30 and the range in which the sensor device 20 irradiates ultrasonic waves may be the same range, or one of them may be wider than the other.

[0023] Figure 2 shows an example of the configuration of the sensor device 20. Figure 2 is a view of the sensor device 20 from below. As shown in Figure 2, the sensor device 20 has a housing 22, a plurality of ultrasonic transmitting elements 23, and a plurality of ultrasonic receiving elements 24. The housing 22 is flattened in the vertical direction and has a rectangular shape when viewed from above (bottom view).

[0024] As an example, the ultrasonic transmitting elements 23 are arranged one in the center and one in each of the four corners of the housing 22. The ultrasonic transmitting elements 23 transmit ultrasonic waves as a transmission wave from above the group of articles W placed on the mounting area R1 toward the mounting area R1, i.e., toward downward. In other words, the ultrasonic transmitting elements 23 transmit ultrasonic waves toward the group of articles W placed on the mounting area R1. The ultrasonic transmitting elements 23 may also be audio speakers. The ultrasonic transmitting elements 23 may also be composed of antenna elements.

[0025] Some modern audio speakers and microphones have frequency response characteristics up to several tens of kHz. Using these, frequencies above 10 kHz, which are difficult for the human ear to perceive, and below 40 kHz, which can reach several meters, are considered ideal for ultrasonic measurement.

[0026] Here, the ultrasonic transmitting element 23 transmits ultrasonic waves in the range of 10 to 40 kHz according to the control of the control device 60.

[0027] The ultrasonic receiving element 24 receives the reflected ultrasonic waves transmitted from the ultrasonic transmitting element 23. Specifically, the ultrasonic receiving element 24 receives the ultrasonic waves (reflected waves) transmitted from the ultrasonic transmitting element 23 and reflected by the group of articles W placed on the mounting area R1, above the group of articles W. That is, the ultrasonic receiving element 24 can receive ultrasonic waves (reflected waves) from below. The ultrasonic transmitting element 23 may be an audio microphone. The ultrasonic receiving element 24 may be composed of an antenna element.

[0028] The number of ultrasonic receiving elements 24 is greater than the number of ultrasonic transmitting elements 23. Multiple ultrasonic transmitting elements 23 and multiple ultrasonic receiving elements 24 are distributed at irregular or equal intervals (for example, a mixture of 11 mm and 14 mm intervals) in a planar area with an area approximately the same size as the mounting area R1. This reduces the appearance of false images.

[0029] Next, the control systems of the control device 60 and the sensor device 20 will be described. Figure 3 is a block diagram showing an example of the control system for the control device 60 and the sensor device 20. As shown in Figure 3, the control device 60 includes a processor 121, a display unit 122, an operation unit 123, a camera controller 124, a mobile device controller 125, a sensor controller 126, a communication unit 127, and a storage unit 128. The processor 121, display unit 122, operation unit 123, camera controller 124, mobile device controller 125, sensor controller 126, communication unit 127, and storage unit 128 are connected to each other via a data bus or the like.

[0030] The processor 121 has the function of controlling the operation of the entire control device 60. The processor 121 may also be equipped with an internal cache and various interfaces. The processor 121 performs various processes by executing programs that are pre-stored in the internal memory or storage unit 128.

[0031] Furthermore, some of the various functions realized by the execution of a program by the processor 121 may be realized by hardware circuits. In this case, the processor 121 controls the functions executed by the hardware circuits.

[0032] For example, processor 121 is a control device for a CPU (Central Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), etc.

[0033] The display unit 122 displays image data from the processor 121. For example, the display unit 122 is composed of a liquid crystal monitor. If the operation unit 123 is composed of a touch panel, the display unit 122 may be formed integrally with the touch panel of the operation unit 123.

[0034] The control unit 123 receives various operation inputs from the operator. The control unit 123 transmits a signal indicating the input operation to the processor 121. For example, the control unit 123 is composed of a mouse, keyboard, or touch panel.

[0035] The camera controller 124 (camera interface) is connected to the camera 30. The camera controller 124 controls the operation of the camera 30 according to the control of the processor 121. For example, the camera controller 124 causes the camera 30 to capture an image. The camera controller 124 also acquires optical distance information from the camera 30 and transmits it to the processor 121.

[0036] The mobile device controller 125 (mobile device interface) is connected to the mobile device 50. The mobile device controller 125 controls the operation of the mobile device 50 according to the control of the processor 121. The mobile device controller 125 controls the operation of the arm portion 112 and the gripping portion 113.

[0037] The sensor controller 126 (sensor interface) is connected to the sensor device 20. The sensor controller 126 controls the operation of the sensor device 20 according to the control of the processor 121. For example, the sensor controller 126 causes the ultrasonic transmitting element 23 to transmit ultrasonic waves. The sensor controller 126 also receives a reflected wave signal (sensing result) indicating the intensity of the received wave from the ultrasonic receiving element 24 and transmits it to the processor 121.

[0038] The communication unit 127 is an interface for sending and receiving data with external devices. The communication unit 127 connects to external devices via a network or the like. For example, the communication unit 127 supports wired or wireless LAN connections.

[0039] The storage unit 128 stores data temporarily or permanently. The storage unit 128 is implemented by, for example, a main memory device such as RAM (Random Access Memory), and an auxiliary memory device such as a semiconductor memory element such as flash memory or a hard disk. The storage unit 128 stores programs or setting information related to the operation of the cargo handling device 10. The storage unit 128 also stores limit information indicating the vertical and horizontal movement limits of the moving device 50, the range of motion of the arm portion 112, and the upper and lower limits of the dimensions of the articles W1 that can be gripped by the gripping portion 113. The storage unit 128 may also store cargo information indicating the size (dimensions) or shape of the articles W1.

[0040] Next, the recovery device 20 will be described. As shown in Figure 3, the sensor device 20 includes an ultrasonic transmitting element 23, an ultrasonic receiving element 24, a transmission control unit 301, an operational amplifier 302, an operational amplifier 305, and a filter 306. The transmission control unit 301 is connected to the sensor controller 126, the operational amplifier 302, and the filter 306. The operational amplifier 302 is connected to the ultrasonic transmitting element 23. The filter 306 is connected to the sensor controller 126 and the operational amplifier 305. The operational amplifier 305 is connected to the ultrasonic receiving element 24.

[0041] The transmission control unit 301 generates a frequency signal corresponding to the frequency of the ultrasonic waves to be transmitted, according to the control from the sensor controller 126. The transmission control unit 301 transmits the generated frequency signal to the ultrasonic transmitting element 23 via the operational amplifier 302. That is, the frequency signal generated by the transmission control unit 301 is amplified via the operational amplifier 302 and then input to the ultrasonic transmitting element 23. More specifically, the transmission control unit 301 sequentially transmits burst signals or chirp signals to multiple ultrasonic transmitting elements 23. As a result, the ultrasonic waves transmitted from the ultrasonic transmitting elements 23 become burst waves or chirp waves. More specifically, the transmission control unit 301 drives the multiple ultrasonic transmitting elements 23 one or more at a time in sequence at a fixed period.

[0042] The ultrasonic transmitting element 23 transmits ultrasonic waves corresponding to the frequency signal. The ultrasonic receiving element 24 receives the reflected ultrasonic waves transmitted by the ultrasonic transmitting element 23. The ultrasonic receiving element 24 transmits a reflected wave signal corresponding to the intensity of the received reflected wave to the filter 306 via the operational amplifier 305. That is, the reflected wave signal from the ultrasonic receiving element 24 is amplified via the operational amplifier 305 and then input to the filter 306.

[0043] Although Figure 3 shows one ultrasonic transmitting element 23 and one ultrasonic receiving element 24, multiple ultrasonic transmitting elements 23 and ultrasonic receiving elements 24 are provided. Similarly, multiple operational amplifiers 302 and 305 are provided, corresponding to the ultrasonic transmitting elements 23 and ultrasonic receiving elements 24.

[0044] The filter 306 performs filtering operations on both the input frequency signal and the reflected wave signal, and outputs a signal to the sensor controller 126 indicating the filtered values ​​of those signals.

[0045] Next, the functions implemented by the control device 60 will be described. The functions implemented by the control device 60 are realized by the processor 121 executing a program stored in the internal memory or storage unit 128, etc.

[0046] First, the processor 121 has a function to acquire optical distance information obtained by photographing the group of objects W using the camera 30. For example, the processor 121 receives a signal through the communication unit 127 or the like indicating that a group of items W has been loaded onto the loading area R1. Upon receiving this signal, the processor 121 causes the camera 30 to take an image via the camera controller 124. The processor 121 also obtains optical distance information from the camera 30 via the camera controller 124.

[0047] The processor 121 may acquire optical distance information when a predetermined operation is input through the operation unit 123. Furthermore, the processor 121 may detect, through other sensors or the like, that a group of items W has been placed on the mounting area R1.

[0048] Furthermore, the processor 121 has the function of measuring the distance to each point in the group of items W using the sensor device 20.

[0049] First, the processor 121 uses one of the ultrasonic transmitting elements 23 to emit ultrasound with a first frequency (for example, 30 kHz). When ultrasound is emitted, the processor 121 receives reflected wave signals (first reflected waves) from each ultrasonic receiving element 24. The processor 121 similarly emits ultrasound with a first frequency using each ultrasonic transmitting element 23 and receives reflected wave signals from each ultrasonic receiving element 24.

[0050] The processor 121 performs a delay summing process based on the reflected wave signal, according to the direction of each point, and calculates the timing at which the intensity of the reflected wave signal at a first frequency exceeds a predetermined threshold. Based on the calculated timing, the processor 121 calculates the distance at each point. The processor 121 generates ultrasonic distance information that associates the position (e.g., coordinates) and distance of each point.

[0051] Furthermore, the processor 121 uses each ultrasonic transmitting element 23 to irradiate ultrasonic waves having a second frequency (for example, 15 kHz) different from the first frequency, and similarly acquires the reflected wave signal of the reflected wave (second reflected wave) at each point.

[0052] The intensity of the reflected wave signal varies depending on the shape of the top surface of the group of items W. Figure 4 shows the surface shapes of the group of items W.

[0053] As shown in Figure 4, the shape of the group of articles W here is flat, convex, concave, or stepped.

[0054] Figure 5 is a graph showing the relationship between shape and the intensity of the reflected wave signal (received intensity). Here, the ultrasonic frequency is assumed to be a predetermined value (for example, 30 kHz). Figure 5 also shows the intensity of the reflected wave signal when the sensor device 20 and the group of objects W are at a predetermined distance apart. In Figure 5, the horizontal axis represents time, and the vertical axis represents the intensity of the reflected wave signal.

[0055] Figure 5 shows graphs 71 to 74. Graph 71 shows the intensity of the reflected wave signal from the plane. Graph 72 shows the intensity of the reflected wave signal from the convex surface. Graph 73 shows the intensity of the reflected wave signal from the concave surface. Graph 74 shows the intensity of the reflected wave signal from the stage.

[0056] The processor 121 may generate ultrasonic distance information based on the reflected wave at a second frequency.

[0057] Furthermore, the processor 121 has the function of estimating the shape of each point in the group of articles W based on the intensity of the reflected wave signal at a first frequency and the intensity of the reflected wave signal at a second frequency.

[0058] Here, we will explain the relationship between ultrasonic frequency and the shape of an object. The reflected wave is given by the following equation.

[0059]

number

[0060] Here, R(t) represents the reflected wave. W(r) is a function representing the shape-dependent reflection of ultrasonic waves. P(t) represents the transmitted wave. Also, as shown in Figure 6, r0 and r1 represent the distance from the ultrasonic transmitting element 23 to the surface (top surface) of the group of items W. As described above, R(t) is obtained by the convolution integral from r0 to r1.

[0061] If the Fourier transforms of W(r), P(t), and R(t) are W(ω), P(ω), and R(ω), then the following equations hold.

[0062]

number

[0063] Figure 7 is a graph showing an example of R(ω). In Figure 7, the horizontal axis represents the ultrasonic frequency, and the vertical axis represents the value of R(ω).

[0064] Figure 7 shows graphs 81 to 84. Graph 81 shows R(ω) in the plane. Graph 82 shows R(ω) on a convex surface. Graph 83 shows R(ω) in a concave surface. Graph 84 shows R(ω) at each stage.

[0065] Figure 8 shows the received signal strength at the first frequency and the received signal strength at the second frequency. As shown in Graph 81, when the shape of the group of items W is planar, both the received signal strength at the first frequency and the received signal strength at the second frequency are large.

[0066] Furthermore, as shown in Graph 82, when the shape of the group of items W is convex, the received signal strength at the first frequency is moderate, and the received signal strength at the second frequency is high.

[0067] Furthermore, as shown in Graph 83, when the shape of the group of articles W is concave, the received signal strength at the first frequency is small, and the received signal strength at the second frequency is large.

[0068] Furthermore, as shown in Graph 84, when the shape of the group of items W is stepped, the received signal strength at the first frequency is high, and the received signal strength at the second frequency is low.

[0069] Based on the above characteristics, the processor 121 estimates the shape of each point on the upper surface of the group of articles W.

[0070] For example, the processor 121 estimates that the shape is planar if both the received signal strength at the first frequency and the received signal strength at the second frequency are above a predetermined threshold. Alternatively, the processor 121 may also estimate that the shape is planar if the difference between the received signal strength at the first frequency and the received signal strength at the second frequency is below a predetermined threshold.

[0071] Furthermore, the processor 121 estimates that the shape is convex if the received signal strength at the first frequency is within a predetermined range and is equal to or greater than a predetermined threshold compared to the received signal strength at the second frequency. The processor 121 may also estimate that the shape is convex if the difference between the received signal strength at the first frequency and the received signal strength at the second frequency is within a predetermined range.

[0072] Furthermore, the processor 121 estimates that the shape is concave if the received signal strength at the first frequency is below a predetermined threshold and the received signal strength at the second frequency is above a predetermined threshold. Alternatively, the processor 121 may also estimate that the shape is concave if the received signal strength at the first frequency is smaller than the received signal strength at the second frequency and the difference between the two is above a predetermined threshold.

[0073] Furthermore, the processor 121 estimates that the shape is stepped if the received signal strength at the first frequency is above a predetermined threshold and the received signal strength at the second frequency is below a predetermined threshold. Alternatively, the processor 121 may also estimate that the shape is stepped if the received signal strength at the first frequency is greater than the received signal strength at the second frequency and the difference between the two is above a predetermined threshold.

[0074] The processor 121 similarly estimates the shape at each point on the upper surface of the group of items W. The processor 121 generates shape information that associates the position (e.g., coordinates) of each point with its shape.

[0075] Figure 9 shows an example of shape information. In Figure 9, the shape of each point is shown when the mounting area R1 is viewed from above. In the example shown in Figure 9, convex surfaces are arranged diagonally in a continuous sequence.

[0076] Furthermore, the processor 121 has the function of recognizing the article W1 based on optical distance information, ultrasonic distance information, and shape information.

[0077] Here, the resolution of ultrasonic distance information and shape information is assumed to be lower than the resolution of optical distance information.

[0078] For example, the processor 121 generates distance information indicating the distance at each point based on optical distance information and ultrasonic distance information. When camera 30 photographs transparent, reflective, or black objects, it may not be able to properly obtain data for areas where the laser beam used for distance measurement does not reflect. In such cases, processor 121 generates distance information by applying the distance indicated by ultrasonic distance information and the shape indicated by shape information to the areas where data could not be properly obtained by camera 30. That is, processor 121 complements the areas where data could not be properly obtained in optical distance information based on ultrasonic distance information and shape information.

[0079] Once distance information is generated, the processor 121 recognizes each item W1 based on the distance information. For example, the processor 121 recognizes the overall shape of the item W1. For example, if the processor 121 recognizes the overall shape as a series of convex surfaces as shown in Figure 9, it recognizes it as a pipe-shaped item.

[0080] Furthermore, if the height of the group of articles W based on ultrasonic distance information and shape information is higher than the height of the group of articles W indicated by optical distance information, the processor 121 may determine that the group of articles W is covered with a film or the like. For example, the processor 121 may determine that the group of articles W is an article in which multiple articles are bound together with a film.

[0081] Furthermore, the method by which the processor 121 recognizes an item is not limited to a specific method.

[0082] Furthermore, the processor 121 may recognize the position, external dimensions, shape, or orientation of an item based on distance information.

[0083] Furthermore, the processor 121 has the function of generating an action plan for grasping the recognized item W1.

[0084] For example, based on the recognition result, the processor 121 sets a route for the gripping unit 113 to move from its current position to a position where it can grip the item W1 and to its destination, as well as the movement speed of the gripping unit 113. The processor 121 generates an operation plan that shows the route, movement speed, etc.

[0085] Furthermore, the processor 121 may generate an operation plan based on ultrasonic distance information and shape information. For example, if the processor 121 determines, based on ultrasonic distance information, that there is a reflective object that reflects ultrasonic waves in an area where nothing exists according to optical distance information, it will generate a route that avoids that area. Also, if the processor 121 determines, based on ultrasonic distance information, that the item W1 is covered with a film or the like, it may reduce the movement speed of the gripping unit 113.

[0086] Furthermore, the method by which the processor 121 generates the operation plan is not limited to a specific method.

[0087] Furthermore, the processor 121 has the function of gripping the article W1 according to the operation plan. The processor 121 moves the gripping unit 113 to a position where it can grip the article W1, according to the operation plan, via the moving device controller 125. Once the gripping unit 113 is moved, the processor 121 uses the gripping unit 113 to grip the article W1. After gripping the article W1, the processor 121 moves the gripping unit 113 to its destination. Once the gripping unit 113 has reached its destination, the processor 121 uses the gripping unit 113 to release the article W1.

[0088] Next, an example of the operation of the control device 60 will be described. Figure 10 is a flowchart illustrating an example of the operation of the control device 60. First, the processor 121 of the control device 60 acquires optical distance information using the camera 30 (S11). Once the optical distance information is acquired, the processor 121 uses the ultrasonic transmitting element 23 to emit ultrasonic waves having a first frequency (S12). After emitting the ultrasonic waves, the processor 121 uses the ultrasonic receiving element 24 to acquire the received intensity at the first frequency (S13).

[0089] When the received signal strength at the first frequency is obtained, the processor 121 uses the ultrasonic transmitting element 23 to emit ultrasound with a second frequency (S14). After emitting the ultrasound, the processor 121 uses the ultrasonic receiving element 24 to obtain the received signal strength at the second frequency (S15).

[0090] Upon obtaining the received signal strength at the second frequency, the processor 121 generates ultrasonic distance information indicating the distance at each point in the ultrasonic irradiation range based on the reflected wave at the first frequency (or second frequency) (S16).

[0091] After calculating the ultrasonic distance information, the processor 121 generates shape information indicating the shape at each point based on the received intensity at the first frequency and the received intensity at the second frequency (S17).

[0092] Once shape information is generated, the processor 121 recognizes the item W1 based on the optical distance information, ultrasonic distance information, and shape information (S18). Once the item W1 is recognized, the processor 121 generates an action plan (S19).

[0093] Once the motion plan is generated, the processor 121 uses the moving device 50 to grasp the item W1 according to the motion plan and release it at the destination (S20). After grasping the item W1 and releasing it at the destination, the processor 121 terminates its operation.

[0094] The processor 121 may repeat steps S12 and S13 and S14 and S15 a predetermined number of times. Furthermore, if the processor 121 recognizes multiple items, it may execute steps S19 and S20 for each item.

[0095] Furthermore, the processor 121 may recognize an object based on ultrasonic distance information and shape information, rather than on optical distance information.

[0096] Furthermore, the processor 121 may refer to the package information to obtain the external dimensions or shape of the recognized item W1.

[0097] The processor 121 may also irradiate the article W1 with ultrasound having three or more different frequencies. In this case, the processor 121 estimates the shape of the top surface of the article W1 based on the received intensity of each ultrasound.

[0098] The cargo handling system configured as described above measures the distance to an item by irradiating it with ultrasound. As a result, the cargo handling system can measure the distance to an item even when the item is transparent or black and therefore cannot be measured using optical information such as lasers or images.

[0099] Furthermore, the cargo handling system irradiates the object with ultrasound at multiple frequencies and measures the received intensity of each ultrasound. Based on the received intensity of each ultrasound, the cargo handling system estimates the shape of the object. As a result, the cargo handling system can effectively recognize the shape of the object. (Second embodiment) Next, a second embodiment will be described. The cargo handling system according to the second embodiment differs from that according to the first embodiment in that it irradiates the goods with ultrasonic waves from multiple directions. Therefore, other parts are denoted by the same reference numerals and detailed descriptions are omitted.

[0100] Figure 1 schematically shows an example configuration of a cargo handling system 1' according to a second embodiment. As shown in Figure 1, the cargo handling system 1' includes a sensor device 20' instead of a sensor device 20. The control device 60 and the sensor device 20' are connected, for example, via a serial cable or LAN.

[0101] Figure 11 shows an example of the configuration of the sensor device 20'. As shown in Figure 11, the sensor device 20' comprises a plurality of ultrasonic transmitting elements 23 and a plurality of ultrasonic receiving elements 24.

[0102] One ultrasonic transmitting element 23 irradiates the article W1 with ultrasonic waves from a predetermined direction. This ultrasonic transmitting element 23 irradiates the article W1 with ultrasonic waves from above at an angle. The other ultrasonic transmitting element 23 irradiates the article W1 with ultrasonic waves from a different direction.

[0103] For example, some ultrasonic transmitting elements 23 irradiate the top surface of article W1 with ultrasound. Other ultrasonic transmitting elements 23 irradiate the side surface of article W1 with ultrasound.

[0104] Furthermore, the processor 121, similar to the first embodiment, uses the ultrasonic transmitting element 23 to irradiate the article W1 with ultrasonic waves having a first frequency and a second frequency. The processor 121 uses the ultrasonic receiving element 24 to acquire the received intensity at the first frequency and the received intensity at the second frequency.

[0105] The processor 121 estimates the shape of the top surface and the shape of the sides of article W1 based on the received signal strength at a first frequency and the received signal strength at a second frequency. The processor 121 generates shape information indicating the estimated shape of the top surface and the shape of the sides. Furthermore, the processor 121 recognizes the item W1 based on the generated shape information and other factors. Furthermore, the processor 121 may generate an operation plan based on the generated shape information, etc.

[0106] The ultrasonic receiving element 24 may be installed at an angle corresponding to the angle at which the ultrasonic transmitting element 23 transmits ultrasonic waves.

[0107] The cargo handling system configured as described above irradiates the sides of the item with ultrasound at multiple frequencies. Therefore, the cargo handling system can also estimate the shape of the item's sides. As a result, the cargo handling system can effectively recognize the shape of the item.

[0108] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0109] 1...Cargo handling system, 1'...Cargo handling system, 2...Pallet, 10...Cargo handling device, 20...Sensor device, 20'...Sensor device, 22...Housing, 23...Ultrasonic transmitting element, 24...Ultrasonic receiving element, 30...Camera, 40...Frame, 41...Lower frame member, 42...Upper frame member, 43...Column member, 50...Moving device, 60...Control device, 71...Graph, 72...Graph, 73...Graph, 74...Graph, 81...Graph, 82...Graph, 83... Graph, 84...Graph, 111...Base, 112...Arm section, 113...Gripping section, 121...Processor, 122...Display section, 123...Operation section, 124...Camera controller, 125...Movement device controller, 126...Sensor controller, 127...Communication section, 128...Storage section, 301...Transmission control section, 302...Operation amplifier, 305...Operation amplifier, 306...Filter, G...Floor surface, R1...Placement area, W...Item group, W1...Item.

Claims

1. A sensor interface connected to an ultrasonic transmitting element that irradiates an object with ultrasonic waves and an ultrasonic receiving element that receives reflected waves from the object, The ultrasonic transmitting element emits ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency via the sensor interface, and the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by the ultrasonic receiving element is acquired. If the intensity of the first reflected wave and the intensity of the second reflected wave are greater than or equal to a predetermined threshold, the shape of the article is estimated to be planar. If the intensity of the first reflected wave is within a predetermined range and the intensity of the second reflected wave is equal to or greater than the predetermined threshold, the shape of the article is presumed to be convex. If the intensity of the first reflected wave is less than or equal to the predetermined threshold, and the intensity of the second reflected wave is greater than or equal to the predetermined threshold, then the shape of the article is presumed to be concave. If the intensity of the first reflected wave is greater than or equal to the predetermined threshold, and the intensity of the second reflected wave is less than or equal to the predetermined threshold, the shape of the article is presumed to be stepped. To generate shape information indicating the estimated shape of the article, Processor and An information processing device equipped with the following features.

2. A sensor interface connected to an ultrasonic transmitting element that irradiates an object with ultrasonic waves and an ultrasonic receiving element that receives reflected waves from the object, The ultrasonic transmitting element emits ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency via the sensor interface, and the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by the ultrasonic receiving element is acquired. If the difference between the intensity of the first reflected wave and the intensity of the second reflected wave is less than or equal to a predetermined threshold, the shape of the article is estimated to be planar. If the difference between the intensity of the first reflected wave and the intensity of the second reflected wave is within a predetermined range, the shape of the article is presumed to be convex. If the intensity of the first reflected wave is less than the intensity of the second reflected wave, and the difference between the two is greater than or equal to the predetermined threshold, the shape of the article is presumed to be concave. If the intensity of the first reflected wave is greater than the intensity of the second reflected wave, and the difference between the two is greater than or equal to the predetermined threshold, then it is estimated that the shape of the article is stepped. To generate shape information indicating the estimated shape of the article, Processor and An information processing device equipped with the following features.

3. The processor generates ultrasonic distance information indicating the distance to the article based on the first reflected wave or the second reflected wave. The information processing apparatus according to claim 1 or 2.

4. The processor recognizes the overall shape of the article based on the ultrasonic distance information and the shape information. The information processing apparatus according to claim 3.

5. It includes a camera interface that connects to a camera that optically measures the distance to the aforementioned article, The aforementioned processor, Using the camera, optical distance information indicating the distance to the article is acquired. The article is further recognized based on the aforementioned optical distance information. The information processing apparatus according to claim 4.

6. The processor determines the region in the optical distance information where data could not be obtained properly. Based on the ultrasonic distance information and the shape information, the article is recognized. The information processing apparatus according to claim 5.

7. It includes a mobile device interface that connects to a mobile device that grips and moves the aforementioned article, The aforementioned processor, Based on the ultrasonic distance information and the shape information, the moving device generates a motion plan for moving the article. The information processing apparatus according to claim 3.

8. The system comprises multiple ultrasonic transmitting elements, The ultrasonic transmitting element is installed to irradiate the top and side surfaces of the article with ultrasonic waves. The processor generates the shape information indicating the shape of the top and side surfaces of the article. The information processing apparatus according to claim 1 or 2.

9. An information processing method performed by a processor, An ultrasonic transmitting element irradiates an article with ultrasonic waves, irradiating it with ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency, and the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by an ultrasonic receiving element are obtained. If the intensity of the first reflected wave and the intensity of the second reflected wave are greater than or equal to a predetermined threshold, the shape of the article is estimated to be planar. If the intensity of the first reflected wave is within a predetermined range and the intensity of the second reflected wave is equal to or greater than the predetermined threshold, the shape of the article is presumed to be convex. If the intensity of the first reflected wave is less than or equal to the predetermined threshold, and the intensity of the second reflected wave is greater than or equal to the predetermined threshold, then the shape of the article is presumed to be concave. If the intensity of the first reflected wave is greater than or equal to the predetermined threshold, and the intensity of the second reflected wave is less than or equal to the predetermined threshold, the shape of the article is presumed to be stepped. To generate shape information indicating the estimated shape of the article, Information processing methods.

10. An information processing method performed by a processor, An ultrasonic transmitting element irradiates an article with ultrasonic waves, irradiating it with ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency, and the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by an ultrasonic receiving element are obtained. If the difference between the intensity of the first reflected wave and the intensity of the second reflected wave is less than or equal to a predetermined threshold, the shape of the article is estimated to be planar. If the difference between the intensity of the first reflected wave and the intensity of the second reflected wave is within a predetermined range, the shape of the article is presumed to be convex. If the intensity of the first reflected wave is less than the intensity of the second reflected wave, and the difference between the two is greater than or equal to the predetermined threshold, the shape of the article is presumed to be concave. If the intensity of the first reflected wave is greater than the intensity of the second reflected wave, and the difference between the two is greater than or equal to the predetermined threshold, then it is estimated that the shape of the article is stepped. To generate shape information indicating the estimated shape of the article, Information processing methods.

11. A program executed by a processor, The aforementioned processor, A function that irradiates an object with ultrasonic waves from an ultrasonic transmitting element, irradiating it with ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency, and acquires the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by an ultrasonic receiving element, If the intensity of the first reflected wave and the intensity of the second reflected wave are greater than or equal to a predetermined threshold, the shape of the article is estimated to be planar. If the intensity of the first reflected wave is within a predetermined range and the intensity of the second reflected wave is equal to or greater than the predetermined threshold, the shape of the article is presumed to be convex. If the intensity of the first reflected wave is less than or equal to the predetermined threshold, and the intensity of the second reflected wave is greater than or equal to the predetermined threshold, then the shape of the article is presumed to be concave. If the intensity of the first reflected wave is greater than or equal to the predetermined threshold, and the intensity of the second reflected wave is less than or equal to the predetermined threshold, the shape of the article is presumed to be stepped. A function to generate shape information indicating the estimated shape of the article, A program that achieves this.

12. A program executed by a processor, The aforementioned processor, A function that irradiates an object with ultrasonic waves from an ultrasonic transmitting element, irradiating it with ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency, and acquires the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by an ultrasonic receiving element, If the difference between the intensity of the first reflected wave and the intensity of the second reflected wave is less than or equal to a predetermined threshold, the shape of the article is estimated to be planar. If the difference between the intensity of the first reflected wave and the intensity of the second reflected wave is within a predetermined range, the shape of the article is presumed to be convex. If the intensity of the first reflected wave is less than the intensity of the second reflected wave, and the difference between the two is greater than or equal to the predetermined threshold, the shape of the article is presumed to be concave. If the intensity of the first reflected wave is greater than the intensity of the second reflected wave, and the difference between the two is greater than or equal to the predetermined threshold, then it is estimated that the shape of the article is stepped. A function to generate shape information indicating the estimated shape of the article, A program that achieves this.

13. A system comprising a sensor device and an information processing device, The aforementioned sensor device is An ultrasonic transmitting element that irradiates an object with ultrasonic waves, An ultrasonic receiving element that receives reflected waves from the aforementioned article and Equipped with, The aforementioned information processing device is A sensor interface connected to the ultrasonic transmitting element and the ultrasonic receiving element, The ultrasonic transmitting element emits ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency via the sensor interface, and the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by the ultrasonic receiving element is acquired. If the intensity of the first reflected wave and the intensity of the second reflected wave are greater than or equal to a predetermined threshold, the shape of the article is estimated to be planar. If the intensity of the first reflected wave is within a predetermined range and the intensity of the second reflected wave is equal to or greater than the predetermined threshold, the shape of the article is presumed to be convex. If the intensity of the first reflected wave is less than or equal to the predetermined threshold, and the intensity of the second reflected wave is greater than or equal to the predetermined threshold, then the shape of the article is presumed to be concave. If the intensity of the first reflected wave is greater than or equal to the predetermined threshold, and the intensity of the second reflected wave is less than or equal to the predetermined threshold, the shape of the article is presumed to be stepped. To generate shape information indicating the estimated shape of the article, Processor and Equipped with, system.

14. A system comprising a sensor device and an information processing device, The aforementioned sensor device is An ultrasonic transmitting element that irradiates an object with ultrasonic waves, An ultrasonic receiving element that receives reflected waves from the aforementioned article and Equipped with, The aforementioned information processing device is A sensor interface connected to the ultrasonic transmitting element and the ultrasonic receiving element, The ultrasonic transmitting element emits ultrasonic waves having a first frequency and ultrasonic waves having a second frequency different from the first frequency via the sensor interface, and the intensity of the first reflected wave of the ultrasonic waves having the first frequency and the second reflected wave of the ultrasonic waves having the second frequency received by the ultrasonic receiving element is acquired. If the difference between the intensity of the first reflected wave and the intensity of the second reflected wave is less than or equal to a predetermined threshold, the shape of the article is estimated to be planar. If the difference between the intensity of the first reflected wave and the intensity of the second reflected wave is within a predetermined range, the shape of the article is presumed to be convex. If the intensity of the first reflected wave is less than the intensity of the second reflected wave, and the difference between the two is greater than or equal to the predetermined threshold, the shape of the article is presumed to be concave. If the intensity of the first reflected wave is greater than the intensity of the second reflected wave, and the difference between the two is greater than or equal to the predetermined threshold, then it is estimated that the shape of the article is stepped. To generate shape information indicating the estimated shape of the article, Processor and Equipped with, system.

Images