Information processing device and information processing method

Abstract

To improve work efficiency in remote control of a construction machine.SOLUTION: An information processing device comprises: an image acquisition part which acquires an image obtained by photographing a subject; an image analysis part which analyzes frequency components of the image and generates high-pass component information indicative of a high-pass component out of the frequency components; a distance information acquisition part which acquires distance information on a distance to the subject; and a determination part which determines whether the subject is flat or not on the basis of the distance information and the high-pass component information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] At civil engineering work sites and the like, construction machines are remotely operated based on images captured by cameras mounted on the machines. The construction machines are, for example, mobile heavy machinery that a person rides in and operates. The captured images are encoded and transmitted to an operator in a remote location. Methods such as H.264 and H.265 are known as video encoding methods. A technology for encoding video that can select an encoding mode with the highest texture reproducibility for an area where texture reproducibility is required is known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-219147 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the amount of code required for captured images becomes large, and a high-bandwidth network is required to transmit the images to a remote location with low latency, which can cause problems when remotely operating construction machinery.

[0005] An object of the present disclosure is to provide an information processing device and an information processing method capable of improving work efficiency in remote operation of construction machinery. [Means for solving the problem]

[0006] The information processing device disclosed herein includes an image acquisition unit that acquires an image of a subject, an image analysis unit that analyzes frequency components of the image and generates high-frequency component information indicating high-frequency components among the frequency components, a distance information acquisition unit that acquires distance information regarding the distance to the subject, and a determination unit that determines whether the subject is flat or non-flat based on the distance information and the high-frequency component information.

[0007] The information processing method disclosed herein includes the steps of acquiring an image of a subject, analyzing frequency components of the image and generating high-frequency component information indicating high-frequency components among the frequency components, acquiring distance information regarding the distance to the subject, and determining whether the subject is flat or non-flat based on the distance information and the high-frequency component information. Effect of the Invention

[0008] According to the present disclosure, it is possible to improve the work efficiency in remotely operating construction machinery. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a remote control system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a method for encoding an image according to the embodiment. [Figure 4] FIG. 4 is a flowchart showing the processing contents of the information processing device according to the embodiment. [Diagram 5] FIG. 5 is a flowchart showing the flow of flatness determination processing for short distances according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the flow of flatness determination processing for a middle distance according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing the flow of flatness determination processing for a long distance according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the present disclosure is not limited to this embodiment, and in the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] [Embodiment] (Remote Control System) A remote control system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a remote control system according to an embodiment.

[0012] 1, the remote control system 1 includes an information processing device 10 and an operation device 12. The information processing device 10 and the operation device 12 are connected via a network N so as to be able to communicate with each other.

[0013] The information processing device 10 is provided on a construction machine arranged at a work site. In this embodiment, the construction machine is not limited to heavy machinery that a person can ride in and move, but also includes fixed machinery that cannot be moved. The construction machine performs, for example, embankment work. The information processing device 10 transmits to the operation device 12 an image that improves the work efficiency in remotely operating the construction machine. In this embodiment, the image is, for example, a moving image.

[0014] The operation device 12 is disposed in an operation room or the like located remotely from the work site. The operation device 12 is equipped with a display unit that displays an image of the work site photographed by the information processing device 10. The operator operates the operation device 12 while checking the image of the work site displayed on the display unit, thereby remotely controlling the construction machine and carrying out embankment work.

[0015] (Information processing device) An example of the configuration of an information processing device according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of an information processing device according to an embodiment.

[0016] As shown in FIG. 2, the information processing device 10 includes a camera 20, a distance sensor 22, a communication unit 24, a storage unit 26, and a control unit .

[0017] The camera 20 captures an image of the periphery of the information processing device 10. The camera 20 captures an image of a subject in the periphery of the information processing device 10. The camera 20 captures, for example, an image of the state of an embankment in the periphery of the information processing device 10 as the subject. The camera 20 outputs the captured image to the image acquisition unit 30.

[0018] The distance sensor 22 measures the distance from the information processing device 10 to the subject. The distance sensor 22 is, for example, a ToF (Time of Flight) sensor. In this case, the distance sensor 22 obtains, for example, a distance image including the distance to the subject. The distance sensor 22 may be, for example, a LiDAR (Light Detection and Ranging).

[0019] The communication unit 24 is a communication interface that performs communication between the information processing device 10 and an external device. The communication unit 24 executes communication between the information processing device 10 and the operation device 14, for example. The communication unit 24 is realized by, for example, a wireless LAN (Local Area Network), Wi-Fi (registered trademark), or the like.

[0020] The storage unit 26 stores various types of information. The storage unit 26 stores information such as the contents of calculations performed by the control unit 28 and programs. The storage unit 26 includes at least one of a RAM (Random Access Memory), a main storage device such as a ROM (Read Only Memory), and an external storage device such as an HDD (Hard Disk Drive).

[0021] The control unit 28 controls each unit of the information processing device 10. The control unit 28 has, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as a RAM or a ROM. The control unit 28 executes a program that controls the operation of the information processing device 10 according to the present invention. The control unit 28 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 28 may be realized by a combination of hardware and software.

[0022] The control unit 28 includes an image acquisition unit 30, a distance information acquisition unit 32, an image analysis unit 34, a determination unit 36, and an encoding unit 38.

[0023] The image acquiring unit 30 controls the camera 20 to capture an image. The image acquiring unit 30 acquires the image captured by the camera 20. The image acquiring unit 30 acquires, for example, an image of a subject captured by the camera 20. The image acquiring unit 30 may remove a DC offset value since the downstream encoding unit 38 targets a signal that does not have a DC offset.

[0024] The distance information acquisition unit 32 controls the distance sensor 22 to measure the distance from the information processing device 10 to the subject. The distance information acquisition unit 32 acquires, from the distance sensor 22, distance information indicating the distance from the information processing device 10 to the subject.

[0025] The image analysis unit 34 analyzes the image acquired by the image acquisition unit 30. For example, the image analysis unit 34 performs frequency analysis on the image acquired by the image acquisition unit 30. For example, the image analysis unit 34 performs a well-known wavelet transform on the image acquired by the image acquisition unit 30. For example, the image analysis unit 34 performs a wavelet transform on the horizontal and vertical directions of the image acquired by the image acquisition unit 30 to decompose the image into components of a plurality of frequency bands. For example, the image analysis unit 34 decomposes the image acquired by the image acquisition unit 30 into low-frequency components, mid-frequency components, and high-frequency components. For example, the image analysis unit 34 generates low-frequency component information indicating the low-frequency components of the image acquired by the image acquisition unit 30, mid-frequency component information indicating the mid-frequency components, and high-frequency component information indicating the high-frequency components. For example, the image analysis unit 34 generates high-frequency component information indicating at least the high-frequency components of the image acquired by the image acquisition unit 30.

[0026] The determination unit 36 ​​determines whether the subject included in the image acquired by the image acquisition unit 30 is flat or non-flat. The determination unit 36 ​​determines whether the embankment included in the image acquired by the image acquisition unit 30 is flat or non-flat. The determination unit 36 ​​determines whether the subject included in the image acquired by the image acquisition unit 30 is flat or non-flat, for example, based on the distance information acquired by the distance information acquisition unit 32 and the high-frequency component information generated by the image analysis unit 34. For example, when a wavelet transform is performed in the horizontal direction of the image and then in the vertical direction, low-frequency components are concentrated in the upper left direction of the output, and high-frequency components are concentrated in the lower right direction. The determination unit 36 ​​can determine whether the surface is flat or non-flat based on the number of high-frequency components. For example, the determination unit 36 ​​divides the image acquired by the image acquisition unit 30 into a plurality of regions and determines whether each region is flat or non-flat.

[0027] The determination unit 36 ​​includes a first determination unit 36a and a second determination unit 36b.

[0028] The first determination unit 36a determines the degree of variation of each pixel of the distance image of the subject included in the distance information acquired by the distance information acquisition unit 32. The first determination unit 36a determines whether the subject is flat or non-flat based on the degree of variation of each pixel of the distance image.

[0029] The second determination unit 36b determines the signal level of the high-frequency component included in the high-frequency component information generated by the image analysis unit 34. The second determination unit 36b determines whether the subject is flat or non-flat based on the signal level of the high-frequency component.

[0030] The determination unit 36 ​​combines the determination results of the first determination unit 36a and the second determination unit 36b to determine whether the subject is flat or non-flat. The determination unit 36 ​​increases the influence of either the determination result of the first determination unit 36a or the determination result of the second determination unit 36b depending on the distance from the information processing device 10 to the subject, which is included in the distance information.

[0031] For example, when the distance from the information processing device 10 to the subject is long, it is difficult to capture the unevenness of the subject from the image, and therefore the signal is less likely to appear in the high-frequency components of the image. That is, the reliability of the distance information acquired by the distance information acquisition unit 32 is higher than the analysis result of the frequency analysis by the image analysis unit 34. In this case, the determination unit 36 ​​increases the influence of the determination result of the first determination unit 36a. Specifically, the determination unit 36 ​​determines that the subject is flat when the degree of variation of each pixel of the distance image is small, even if the signal level of the high-frequency components is high.

[0032] For example, when the distance from the information processing device 10 to the subject is short, the unevenness of the subject is easily captured from the image, and signals are more likely to appear in the high-frequency components of the image. That is, the reliability of the analysis result of the frequency analysis by the image analysis unit 34 is higher than the distance information acquired by the distance information acquisition unit 32. In this case, the determination unit 36 ​​increases the influence of the determination result of the second determination unit 36b. Specifically, the determination unit 36 ​​determines that the subject is flat when the signal level of the high-frequency components is small even if the degree of variation of each pixel in the distance image is large.

[0033] For example, when the distance from the information processing device 10 to the subject is medium, the determination unit 36 ​​determines that the subject is flat if the signal level of the high frequency components is low and the degree of variation in each pixel of the distance image is small.

[0034] Note that the respective threshold values ​​for determining whether the distance from the information processing device 10 to the subject is a short distance, a medium distance, or a long distance may be set in advance.

[0035] The encoding unit 38 encodes the image acquired by the image acquisition unit 30. When the subject included in the image acquired by the image acquisition unit 30 is flat, the encoding unit 38 encodes the image so that the amount of information is low, and when the subject is non-flat, the encoding unit 38 encodes the image so that the amount of information is high. When the image acquired by the image acquisition unit 30 is divided into a plurality of regions, the encoding unit 38 encodes the non-flat regions so that the amount of information is high, and encodes the flat regions so that the amount of information is low. The encoding unit 38 bit-shifts the wavelet transform coefficients of the non-flat regions of the image acquired by the image acquisition unit 30 to the most significant bit side, and encodes the bits resulting from this bit shift by complementing them with 0.

[0036] FIG. 3 is a diagram showing an example of a method for encoding an image according to the embodiment. In FIG. 3, the horizontal axis indicates flat regions and non-flat regions, and the vertical axis indicates bit positions. FIG. 3 shows a schematic diagram of flat region and non-flat region wavelet transform coefficients included in an image acquired by the image acquisition unit 30, which are expanded into bit planes. The most significant bit (MSB) indicates the most significant bit of the flat region. The least significant bit (LSB) indicates the least significant bit of the flat region. As shown in FIG. 3, the encoding unit 38 bit-shifts the wavelet coefficients of the non-flat region to a higher bit side than the most significant bit of the flat region, and encodes the image acquired by the image acquisition unit 30. That is, the encoding unit 38 can prevent the image quality of the non-flat region from deteriorating due to the non-flat region being preferentially encoded from the higher bits.

[0037] (Processing contents) The processing contents of the information processing device according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the processing contents of the information processing device according to the embodiment.

[0038] The control unit 28 is initialized before starting the process (step S10), and then the process proceeds to step S12.

[0039] The control unit 28 sets a signal level threshold value (step S12). Specifically, the control unit 28 sets a signal level threshold value for the high frequency components of the image, which is used to determine whether the subject is flat or non-flat. Then, the process proceeds to step S14.

[0040] The control unit 28 sets a distance threshold value (step S14). Specifically, the control unit 28 sets a threshold value of the distance from the information processing device 10 to the subject, which is used to determine whether the subject is flat or non-flat. Then, the process proceeds to step S16.

[0041] The image acquiring unit 30 acquires an image of the subject from the camera 20 (step S16). Specifically, the image acquiring unit 30 controls the camera 20 to capture an image of the subject, and acquires the image captured by the camera 20. Then, the process proceeds to step S18.

[0042] The distance information acquisition unit 32 acquires distance information indicating the distance from the information processing device 10 to the subject from the distance sensor 22 (step S18). Specifically, the distance information acquisition unit 32 controls the distance sensor 22 to detect the distance from the information processing device 10 to the subject, and acquires distance information indicating the distance detected by the distance sensor 22. Then, the process proceeds to step S20.

[0043] The distance information acquisition unit 32 determines whether the distance from the information processing device 10 to the subject is a short distance (step S20). If it is determined that the distance from the information processing device 10 to the subject is a short distance (step S20; Yes), the process proceeds to step S22. If it is not determined that the distance from the information processing device 10 to the subject is a short distance (step S20; No), the process proceeds to step S24.

[0044] If it is determined as Yes in step S20, the determination unit 36 ​​executes a flatness determination process for a short distance (step S22). Here, the flow of the flatness determination process for a short distance according to the embodiment will be described with reference to the flowchart shown in FIG.

[0045] The first determination unit 36a determines whether the signal level of the high-frequency component of the image of the subject is equal to or lower than the threshold value (step S40). If it is determined that the signal level of the high-frequency component of the image of the subject is equal to or lower than the threshold value (step S40; Yes), the process proceeds to step S44. If it is not determined that the signal level of the high-frequency component of the image of the subject is equal to or lower than the threshold value (step S40; No), the process proceeds to step S42.

[0046] If the result of the determination in step S40 is No, the second determination unit 36b determines whether or not the distance difference between the pixels of the attention area of ​​the subject included in the distance image and the pixels of the peripheral area of ​​the attention area is within a predetermined range (step S42). The attention area is an area including one or more pixels of the subject in the distance image. The attention area is, for example, 2×2=4 pixels. A plurality of attention areas may be set in the subject, or only one may be set. The predetermined range may be set arbitrarily. If the distance difference between the pixels of the attention area of ​​the subject and the pixels of the peripheral area of ​​the attention area is determined to be within the predetermined range (step S42; Yes), the process proceeds to step S44. If the distance difference between the pixels of the attention area of ​​the subject and the pixels of the peripheral area of ​​the attention area is not determined to be within the predetermined range (step S42; No), the process proceeds to step S46.

[0047] If it is determined that the answer is Yes in step S40 or Yes in step S42, the determination unit 36 ​​determines that the subject is flat (step S44), and then ends the process in FIG. 5, and proceeds to step S30 shown in FIG.

[0048] If the result of the determination in step S42 is No, the determination unit 36 ​​determines that the subject is non-flat (step S46), and then ends the process in FIG. 5, and proceeds to step S30 shown in FIG.

[0049] Returning to the explanation using FIG. 4, if it is determined No in step S20, the distance information acquisition unit 32 determines whether or not the distance from the information processing device 10 to the subject is a medium distance (step S24). If it is determined that the distance from the information processing device 10 to the subject is a medium distance (step S24; Yes), the process proceeds to step S26. If it is not determined that the distance from the information processing device 10 to the subject is a medium distance (step S24; No), the process proceeds to step S28.

[0050] If it is determined as Yes in step S24, the determination unit 36 ​​executes flatness determination processing for a medium distance (step S26). Here, the flow of the flatness determination processing for a medium distance according to the embodiment will be described with reference to the flowchart shown in FIG.

[0051] The first determination unit 36a determines whether the signal level of the high-frequency component of the image of the subject is equal to or lower than the threshold value (step S50). If it is determined that the signal level of the high-frequency component of the image of the subject is equal to or lower than the threshold value (step S50; Yes), the process proceeds to step S52. If it is not determined that the signal level of the high-frequency component of the image of the subject is equal to or lower than the threshold value (step S50; No), the process proceeds to step S56.

[0052] If the result of the determination in step S50 is Yes, the second determination unit 36b determines whether or not the distance difference between the pixels of the attention region of the subject included in the distance image and the pixels of the peripheral region of the attention region is within a predetermined range (step S52). If the distance difference between the pixels of the attention region of the subject and the pixels of the peripheral region of the attention region is determined to be within the predetermined range (step S52; Yes), the process proceeds to step S54. If the distance difference between the pixels of the attention region of the subject and the pixels of the peripheral region of the attention region is not determined to be within the predetermined range (step S52; No), the process proceeds to step S56.

[0053] If it is determined as Yes in step S52, the determination unit 36 ​​determines that the subject is flat (step S54), and then ends the process in FIG. 6, and proceeds to step S30 shown in FIG.

[0054] If the result of the determination in step S50 or the result of the determination in step S52 is No, the determination unit 36 ​​determines that the subject is non-flat (step S56), and then ends the process in FIG. 6, and proceeds to step S30 shown in FIG.

[0055] Returning to the explanation using Fig. 4, if the answer is No in step S24, the judgment unit 36 ​​executes flatness judgment processing for long distances (step S28). Here, the flow of the flatness judgment processing for long distances according to the embodiment will be explained using the flowchart shown in Fig. 7.

[0056] The second determination unit 36b determines whether the distance difference between the pixels of the attention region of the subject included in the distance image and the pixels of the peripheral region of the attention region is within a predetermined range (step S60). If it is determined that the distance difference between the pixels of the attention region of the subject and the pixels of the peripheral region of the attention region is within the predetermined range (step S60; Yes), the process proceeds to step S64. If it is not determined that the distance difference between the pixels of the attention region of the subject and the pixels of the peripheral region of the attention region is within the predetermined range (step S60; No), the process proceeds to step S62.

[0057] If the result of the determination in step S60 is No, the first determination unit 36a determines whether or not the signal level of the high-frequency component of the image of the subject is equal to or lower than the threshold value (step S62). If the signal level of the high-frequency component of the image of the subject is equal to or lower than the threshold value (step S62; Yes), the process proceeds to step S64. If the signal level of the high-frequency component of the image of the subject is not equal to or lower than the threshold value (step S62; No), the process proceeds to step S66.

[0058] If the determination is Yes in step S60 or Yes in step S62, the determination unit 36 ​​determines that the subject is flat (step S64), and the process proceeds to step S30 shown in FIG.

[0059] If the result of the determination in step S62 is No, the determination unit 36 ​​determines that the subject is non-flat (step S66), and the process proceeds to step S30 shown in FIG.

[0060] The encoding unit 38 encodes the image acquired by the image acquisition unit 30 (step S30). Specifically, the encoding unit 38 encodes the image so that the amount of information in flat regions is small and the amount of information in non-flat regions is large, based on the determination result of the determination unit 36. Then, the process proceeds to step S32.

[0061] The encoding unit 38 transmits the encoded image to the operation device 12 via the communication unit 24 (step S32). Then, the process of FIG. 4 ends.

[0062] As described above, in this embodiment, an image is encoded so that the amount of information on flat areas where work has been completed is small and the amount of information on non-flat areas where work has not been completed is large at the work site, and the encoded image is transmitted to the operation device 12 operated by the operator of the heavy equipment at a remote location. This allows the operator to operate the heavy equipment based on a clear image of the work site, thereby improving work efficiency.

[0063] Each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. This distribution and integration configuration may be performed dynamically.

[0064] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the contents of these embodiments. The above-mentioned components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-mentioned components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the above-mentioned embodiments. [Explanation of symbols]

[0065] 1 Remote Control System 10. Information processing device 12 Operating device 20 Camera 22 Distance Sensor 24 Communications Department 26 Memory section 28 Control Unit 30 Image acquisition unit 32 Distance information acquisition section 34 Image Analysis Unit 36 Judgment section 36a 1st judgment part 36b Second judgment part 38 Encoding section

Claims

1. an image acquisition unit that acquires an image of a subject; an image analysis unit that analyzes frequency components of the image and generates high-frequency component information indicating high-frequency components among the frequency components; a distance information acquisition unit that acquires distance information relating to a distance to the subject; a determination unit that determines whether the subject is flat or non-flat based on the distance information and the high-frequency component information; An information processing device comprising:

2. the determination unit includes a first determination unit that determines whether the object is flat or non-flat based on a degree of variation in distance indicated by each pixel of a distance image of the object included in the distance information, and a second determination unit that determines whether the object is flat or non-flat based on a signal level of the high-frequency component included in the high-frequency component information, the determination unit determines whether the subject is flat or non-flat by combining determination results of the first determination unit and the second determination unit in accordance with a distance to the subject included in the distance information. The information processing device according to claim 1 .

3. the determination unit increases an influence of the determination result of the first determination unit as the distance to the subject increases, and increases an influence of the determination result of the second determination unit as the distance to the subject decreases. The information processing device according to claim 2 .

4. An encoding unit that encodes the image, The determination unit determines whether the image is flat or non-flat for each of a plurality of regions, the encoding unit encodes the image so that a non-flat region among the plurality of regions has a high amount of information, and encodes the image so that a flat region has a low amount of information. The information processing device according to claim 1 .

5. acquiring an image of a subject; A step of analyzing frequency components of the image and generating high-frequency component information indicating high-frequency components among the frequency components; obtaining distance information relating to a distance to the object; determining whether the object is flat or non-flat based on the distance information and the high frequency component information; An information processing method comprising:

Images