Communication system and control device

Abstract

To provide a communication system between an unmanned vehicle and a control device, capable of improving stability of a moving unmanned vehicle.SOLUTION: A communication system simultaneously executes first wireless communication and second wireless communication, the first wireless communication being configured to transmit an image signal captured by a camera mounted on an unmanned vehicle to a control device through unidirectional communication with a first frequency band, the second wireless communication being configured to transmit control signals from the control device to the unmanned vehicle and transmit telemetry data from the unmanned vehicle to the control device through unidirectional communication with a second frequency band lower than the first frequency band. In the first wireless communication, the image signal is transmitted to the control device in a non-compressed state. The control device selects one image signal based on evaluation values for a receiving state, by block comprising predetermined multiple pixels, for the image signals received by a plurality of receiving antennas, and maps the selected image signal in a picture frame corresponding to one frame.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a communication system for use in an unmanned vehicle and a control device, and to the control device. [Background technology]

[0002] In recent years, unmanned vehicles such as drones have been used in various fields, and communication technologies between unmanned vehicles and the ground have also been developed.

[0003] As for the applications of unmanned vehicles, for example, unmanned aerial vehicles such as drones are being considered for use in inspection, monitoring, disaster response, and the like. In these cases, it may be required to fly over a long distance that is out of the operator's line of sight, for example, to a location about 5 km away. In order to deal with cases where the unmanned vehicle cannot be seen, images captured by a camera mounted on the unmanned vehicle are transmitted to a display device on the operator's side. This allows the operator to operate the unmanned vehicle while viewing the captured images.

[0004] As related prior art, the following Patent Documents 1 and 2 can be mentioned. The following Patent Document 1 discloses that in an unmanned aerial vehicle (UAV) control system, remote operation data and remote sensing data are communicated in different frequency bands (see paragraphs

[0012] and

[0055] , etc.). In addition, the following Patent Document 2 discloses that an image from an on-board camera is transmitted from a vehicle to a server, and that the vehicle is remotely controlled from the server (paragraph

[0043] ). It also discloses that a communication unit of a vehicle may be capable of diversity reception (paragraph

[0071] ). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-169532 A [Patent Document 2] JP 2022-154645 A Summary of the Invention [Problem to be solved by the invention]

[0006] Here, when transmitting an image wirelessly, the image is generally transmitted in a data-compressed state. However, when transmitting compressed images wirelessly, in a situation where the radio wave reception condition gradually deteriorates, such as when an unmanned moving object gradually moves away, the receiving side may at some point become unable to decode the image, and the displayed image may suddenly cease. This is due to the characteristics of the compressed image decoding process, that when the amount of data loss due to a reception error exceeds a certain amount, it becomes impossible to properly decode one image, resulting in a decoding error. When a decoding error occurs, since there is no image to be displayed, a specified error image, such as an all-black image, is displayed, and the display of the captured image is suddenly interrupted.

[0007] When an unmanned vehicle is located too far away to be seen with the naked eye, if the image display suddenly stops as described above, it becomes impossible to properly control the unmanned vehicle. Furthermore, if the image display suddenly stops, it becomes difficult for the pilot or other user to determine whether the cause is a deterioration in radio wave conditions or a malfunction due to the unmanned vehicle crashing or the like, which places a mental burden on the user. For these reasons, the situation in which the image display is suddenly interrupted as described above may hinder the stable movement of the unmanned vehicle.

[0008] The present invention has been made in consideration of the above circumstances, and has an object to realize a communication system between an unmanned vehicle and a control device that can improve the movement stability of the unmanned vehicle. [Means for solving the problem]

[0009] The communication system of the present invention is a communication system between an unmanned mobile body and a control device, which simultaneously executes a first wireless communication in which an image signal captured by a camera attached to the unmanned mobile body is transmitted to the control device by one-way communication in a first frequency band, and a second wireless communication in which a control signal is transmitted from the control device to the unmanned mobile body and telemetry data is transmitted from the unmanned mobile body to the control device by simplex communication in a second frequency band lower than the first frequency band, and in the first wireless communication, the image signal is transmitted to the control device in an uncompressed state, and the control device receives the image signals by multiple receiving antennas and selects one image signal from the image signals received by the multiple receiving antennas based on an evaluation value of the reception state in block units consisting of a predetermined number of pixels, and maps the selected image signal within an image frame for one frame. As described above, the image signal is transmitted in an uncompressed state, the uncompressed image signal is received by a plurality of receiving antennas, and one image signal is selected from the image signals received by the antennas based on an evaluation of the reception state for each predetermined block, and the selected image signal for each block is mapped to one image. As long as there is no reception error for the image signals of all blocks of the image, it is possible to continue displaying the image for at least a portion of the blocks. In other words, it is possible to avoid the occurrence of a situation in which the image display suddenly stops due to the deterioration of the reception state, as in the case of decoding and displaying a compressed image.

[0010] In addition, the control device of the present invention simultaneously executes a first wireless communication, which is a one-way communication in a first frequency band, and a second wireless communication, which is a simplex communication in a second frequency band lower than the first frequency band, with an unmanned mobile body, receives an image signal captured by a camera attached to the unmanned mobile body and transmitted in an uncompressed state by a transmitting device mounted on the unmanned mobile body through the first wireless communication, and transmits a control signal to the unmanned mobile body and receives telemetry data from the unmanned mobile body through the second wireless communication.The control device is equipped with a ground station, a plurality of first communication antennas used to receive the first wireless communication, and a second communication antenna used to transmit and receive the second wireless communication, and the ground station has an image mapping unit that selects one image signal based on an evaluation value of the reception state for each block consisting of a predetermined number of pixels from the image signals received by the plurality of first communication antennas, and maps the selected image signal within an image frame for one frame. This makes it possible to continue displaying images for at least some of the blocks as long as there are no reception errors in the image signals for all blocks of the image, which makes it possible to avoid the occurrence of a situation in which image display suddenly stops due to deterioration of the reception conditions, as occurs when a compressed image is decoded and displayed. Effect of the Invention

[0011] According to the present invention, it is possible to realize a communication system between an unmanned vehicle and a control device, which can improve the movement stability of the unmanned vehicle. [Brief description of the drawings]

[0012] [Figure 1] 1 is an explanatory diagram of a communication system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a perspective view of the control device according to the embodiment. [Diagram 3] FIG. 2 is a perspective view of the embodiment in a state where the control device and the tablet PC are separated. [Figure 4] FIG. 2 is a plan view of the control device according to the embodiment. [Diagram 5]FIG. 2 is a bottom view of the control device according to the embodiment. [Figure 6] FIG. 2 is a perspective view of the control device according to the embodiment with the antenna open. [Figure 7] FIG. 2 is a perspective view of the control device according to the embodiment with the antenna open. [Figure 8] FIG. 2 is a side view of the control device according to the embodiment with the antenna open. [Figure 9] FIG. 2 is a block diagram of each component constituting the communication system of the embodiment. [Figure 10] FIG. 2 is a block diagram of a control device according to the embodiment. [Figure 11] FIG. 2 is a block diagram showing an example of a configuration for implementing image synthesis processing according to an embodiment. [Figure 12] FIG. 2 is a functional block diagram for explaining functions of an image synthesis unit in the embodiment. [Figure 13] 1 is a diagram illustrating an example of a relationship between blocks and small regions according to an embodiment. [Figure 14] FIG. 13 is a diagram showing an example of a mapping image in which a missing portion occurs. [Figure 15] 1 is a flowchart showing a process according to an embodiment. [Figure 16] 11A to 11C are diagrams for explaining the effect of the image synthesis process according to the embodiment. [Figure 17] 16A and 16B are diagrams for explaining the effect of the image synthesis processing according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described. In the following explanation, the control device 1 is referred to as being in the position when the user holds the control device 1 in front of his or her body, with the side with the controls and display screen being the top and the direction the user is facing being the front, and the directions up, down, front, back, left and right are indicated.

[0014] <1. System Overview> Figure 1 shows a drone 2 as an example of an unmanned mobile body, and a control device 1 for the drone 2. The communication system between the control device 1 and the drone 2 is a high-speed wireless transmission system capable of long-distance communication up to about 5 km.

[0015] A control signal is transmitted from the control device 1 to the drone 2 to control the flight of the drone 2 and the shooting operation of the camera 82 mounted on the drone 2. Image signals captured by the camera 82 and telemetry data from the drone 2 are transmitted from the drone 2 to the control device 1.

[0016] In this case, the first wireless communication and the second wireless communication are performed simultaneously in parallel between the control device 1 and the drone 2. In the first wireless communication, an image signal is transmitted from the drone 2 to the control device 1 via one-way communication using a first frequency band, for example the 5.7 GHz band. In the second wireless communication, control signals and telemetry data request signals are transmitted from the control device 1 to the drone 2, and telemetry data is transmitted from the drone 2 to the control device 1, using simplex communication in a second frequency band, for example the sub-GHz band.

[0017] The sub-GHz band of the second wireless communication is, for example, 920 MHz. The telemetry data is information obtained by various sensors on the drone side, such as altitude information, location information by GNSS (Global Navigation Satellite System), temperature information, air pressure information, attitude information of the drone 2, remaining battery information, and the like. Control signals sent from the control device 1 via the second wireless communication include control signals (control signals) for flight control of the drone 2, and control signals (camera operation signals) related to the shooting operation of the camera 82 mounted on the drone 2.

[0018] The image signal transmitted by the first wireless communication is, for example, a Full HD (Full High Definition) image, allowing high-quality images (video or still images) captured by the camera 82 of the drone 2 to be viewed on the control device 1 side.

[0019] The control device 1, which will be described in detail later, is used by attaching a separate information processing device, a tablet PC 3. The display screen of the tablet PC 3 can be used to display images based on received image signals and information based on telemetry data.

[0020] Furthermore, the control device 1 can be connected to, for example, an external monitor device 4, and can be supplied with and display an image signal received via the first wireless communication. In addition, a memory card 5, for example, can be attached to the control device 1 as a storage medium, and image signals received via the first wireless communication can be stored in the memory card 5.

[0021] <2. Structure of the control device> The structure of the control device 1 will be described with reference to FIGS.

[0022] The tablet PC 3 is detachably attached to the main body 10 of the control device 1. Figures 2 and 4 show the tablet PC 3 attached to the main body 10, and Figure 3 shows the tablet PC 3 removed from the control device 1.

[0023] The tablet PC 3 has a display screen 30 formed on the top surface of a substantially rectangular plate-like body, and a terminal section 31 (see FIG. 3) is provided on the right side surface. Accordingly, an arrangement space 15 is formed in the main body 10 of the control device 1, and the tablet PC 3 is attached to the arrangement space 15.

[0024] In this case, the hold bar 12 is rotatably disposed on the upper side of the arrangement space 15. The hold bar 12 is bent to match the shape of the tablet PC 3, and both ends are inserted into the shaft portion 13. This allows the hold bar 12 to be rotated upward from the state shown in FIG. 3. The tablet PC 3 can be inserted into the arrangement space 15 by rotating the hold bar 12 upward.

[0025] Thereafter, the hold bar 12 is returned to its original state, so that the hold bar 12 holds the tablet PC 3 down so as not to fall off, as shown in Fig. 2. In this case, the hold bar 12 is locked by a bar fixing portion 14 formed on the rear end side, so that the hold bar 12 cannot rotate and can stably hold the tablet PC 3.

[0026] The terminal unit 31 of the tablet PC 3 is provided with, for example, a USB (Universal Serial Bus) terminal, and this USB terminal is used for communication with the main body unit 10. The main body 10 is provided with an openable / closable cover 16, and opening the cover 16 enables a cable connection to the terminal unit 31 of the tablet PC 3. For example, a USB cable from inside the main body 10 is disposed inside the cover 16, and with the tablet PC 3 attached, the USB cable is connected to the USB terminal of the terminal unit 31. This brings the controller 1 and the tablet PC 3 into a connected state that enables data transmission.

[0027] A handle 11 is provided at the front end of the main body 10, and a user can carry the control device 1 by using the handle 11. In addition, strap holders 20 are provided on the left and right front ends and the left and right rear ends of the main body 10, and by attaching straps to these, the user can position the control device 1 stably in front of their body.

[0028] The main body 10 is provided with various controls that can be operated by the user. As shown in Fig. 3 and Fig. 4, six operation switches 17 are provided on the left and right sides of the top surface of the main body 10. The operation switches 17 are, for example, three-position alternate switches. The user can assign any operation function to each operation switch 17. For example, a shutter operation or zoom operation for the camera 82 may be assigned, or an operation related to piloting the drone 2 may be assigned.

[0029] In addition, for example, three joysticks 18 are provided on the top surface of the main body 10. The joysticks 18 are operable as sticks and also function as push switches that can be pressed in. For example, the joystick 18 on the front right side of the top surface is used as an operator for operating the gimbal of the camera 82, and the other pair of left and right joysticks 18 are used as operators for piloting the drone 2.

[0030] In addition, a power switch 19 is provided on the top surface of the main body 10. As shown in FIG. 5, switches 27 are provided on the left and right sides of the bottom surface of the main body 10.

[0031] 7 and 8, a charging port 26 is provided on the left side surface of the main body 10. By connecting an external charging device (such as a power adapter) to the terminal inside this charging port 26, it becomes possible to operate the device using an external power source and charge the internal battery.

[0032] The main body 10 is provided with a plurality of antennas. As shown in FIG. 5, dipole antennas 21, 22, and 23 are provided on the lower surface side of the main body 10, and patch antennas 24 and 25 are also provided. The dipole antennas 21, 22, and 23 are rotatable about shafts 21a, 22a, and 23a, respectively. The patch antennas 24 and 25 are also rotatable about shafts 24a and 25a, respectively.

[0033] FIG. 5 shows a state in which the antennas (21 to 25) are stored on the underside of the main body 10. FIG. 6 shows a state in which each of the antennas (21 to 25) is rotated approximately 90 degrees from the stored state. FIG. 7 shows a state in which each of the antennas (21...25) is rotated by 90 degrees or more from the stored state. FIG. 8 shows the state of FIG. 6 from the side of the main body 10. In FIG.

[0034] The dipole antennas 21, 22, and 23 and the patch antennas 24 and 25 can be manually rotated as desired, allowing the user to adjust the reception conditions of image signals and telemetry data by moving each antenna. During system operation, the user deploys each antenna (21...25) as shown in Fig. 6 or Fig. 7 and uses the control device 1.

[0035] The direction in which radio waves are strong can be adjusted by adjusting the angle of each antenna (21...25). For example, the direction in which radio waves are strong is shown by the dashed arrow in Fig. 6. In particular, for patch antennas 24 and 25, the direction of the receiving surface can be changed by adjusting the angle. For example, by adjusting the antenna angle so that the receiving surface is oriented toward drone 2, good communication conditions can be maintained and communication can be performed.

[0036] <3. Transmission and reception configuration> The internal configuration of the drone 2 and the control device 1 will be described with reference to FIG. The drone 2 is equipped with an aerial station 80, a camera 82, a flight controller 83, and antennas 84 and 85.

[0037] The aerial station 80 communicates with the pilot device 1 and processes information transmission to a camera 82 and a flight controller 83 based on the communication. The flight controller 83 controls the flight of the drone 2 and performs sensing using various sensors. The camera 82 captures an image and outputs an image signal SV0. The camera 82 is provided with a gimbal mechanism (not shown).

[0038] The antenna 84 is a first communication antenna used for the first wireless communication, which is a one-way communication in the 5.7 GHz band. The antenna 85 is a second communication antenna used for a second wireless communication, which is simplex communication in the sub-GHz band.

[0039] The aerial station 80 supplies the control signal CM transmitted from the controller 1 by the second wireless communication to the camera 82 (or gimbal mechanism). This causes the camera operation or gimbal operation to be performed in response to the user's operation on the controller 1 side. The aerial station 80 also supplies the control signal CM and the request signal transmitted from the control device 1 through the second wireless communication to the flight controller 83. This allows the drone 2 to fly in response to the user's operation on the control device 1 side, and transmits telemetry data TM in response to the request signal.

[0040] The aerial station 80 also performs a process of transmitting an image signal SV0 captured by the camera 82 from the antenna 84 via the first wireless communication. At this time, the aerial station 80 performs a process of transmitting the image signal SV0 from the antenna 84 via the first wireless communication in an uncompressed state without performing data compression processing. The aerial station 80 also performs a process of transmitting the telemetry data TM input from the flight controller 83 from the antenna 85 via a second wireless communication.

[0041] The control device 1 includes a ground station 50, a memory unit 70, an operation unit 75, and the dipole antennas 21, 22, and 23, and patch antennas 24 and 25 described above.

[0042] In the pilot control device 1, of the three dipole antennas 21, 22, and 23, the dipole antenna 23 at the center of the underside of the main body 10 is used as a second communication antenna used for the second wireless communication. In addition, the dipole antennas 21, 22 and the patch antennas 24, 25 are used as a first communication antenna used for the first wireless communication.

[0043] The storage unit 70 receives the memory card 5 shown in FIG.

[0044] The operation unit 75 is made up of operators such as the operation switch 17, the joystick 18, the power switch 19, and the switch 27, and operation detection circuits for these.

[0045] The ground station 50 receives image signals using a first communication antenna (dipole antennas 21, 22 and patch antennas 24, 25) and receives telemetry data TM using a second communication antenna (dipole antenna 23), and transmits control signals CM and request signals.

[0046] The ground station 50 can transmit the received image signal SV1 to the tablet PC 3 via the terminal 71. The ground station 50 can also transmit the received telemetry data TM to the tablet PC 3 via the terminal 72 . As a result, on the tablet PC 3 side, an application program corresponding to the pilot device 1 is started, so that the input image signal SV1 and telemetry data TM can be processed and displayed on the display screen 30. Therefore, the user can pilot the drone 2 and operate the camera while viewing the images captured by the drone 2 and the telemetry data on the display screen 30.

[0047] Moreover, the ground station 50 can supply the received image signal SV2 to the storage unit 70 so that it can be stored in the memory card 5. Furthermore, the ground station 50 can supply the received image signal SV3 to a terminal 73. When the monitor device 4 is connected to the terminal 73, the image captured by the drone 2 is displayed on the monitor device 4.

[0048] A detailed configuration of the ground station 50 is shown in FIG. The earth station 50 is provided with receiving circuits 51, 52, 54, and 55 corresponding to the dipole antennas 21, 22 and patch antennas 24, 25, respectively, and performs radio wave receiving processing corresponding to each antenna.

[0049] The reception outputs of the receiving circuits 51, 54 of the dipole antenna 21 and the patch antenna 24 are input to a radio wave selection section 56, and the one with the better reception condition is selected.

[0050] The reception outputs of the receiving circuits 52, 55 of the dipole antenna 22 and the patch antenna 25 are input to a radio wave selection section 57, and the one with the better reception condition is selected.

[0051] In the ground station 50 of the embodiment, the output information of each of the radio wave selection units 56 and 57 is temporarily stored in a memory unit 62, the use of which will be described later.

[0052] The signals selected by the radio wave selection units 56 and 57 are supplied to an image synthesis unit 58 using an FPGA (field-programmable gate array) for image synthesis. The image synthesis here is performed by selecting the image signal with the better reception condition for each frame of the image signal in units of a predetermined number of pixels. Here, the image synthesis unit 58 performs image synthesis processing based on the information stored in the memory unit 62 described above, and details of the image synthesis processing as an embodiment performed by the image synthesis unit 58 will be described again later.

[0053] The image signal for each frame synthesized by the image synthesis unit 58 is subjected to necessary processing such as adding an on-screen display (OSD) and changing the resolution by the image setting unit 59, and is output as image signals SV1, SV2, and SV3. For example, the image signal SV1 is supplied to the tablet PC 3 from a terminal 71.

[0054] A CPU (Central Processing Unit) 60 controls the synthesis process of the image synthesis unit 58 and the OSD process of the image setting unit 59 in relation to the above-mentioned image signal reception.

[0055] A transmission / reception circuit 53 and a modulation / demodulation unit 61 are provided for the dipole antenna 23. For example, when transmitting a control signal or a request signal in response to an operation of the operation unit 75, the CPU 60 supplies data such as these control signals to the modulation / demodulation unit 61. This data such as the control signal is modulated by the modulation / demodulation unit 61 and transmitted from the dipole antenna 23 via the transmission / reception circuit 53.

[0056] Moreover, the radio waves received by the dipole antenna 23 are detected by the transmission / reception circuit 53 and demodulated by the modulation / demodulation unit 61. As a result, the telemetry data TM from the drone 2 is received and transmitted to the tablet PC 3 via the terminal 72. In this example, the above-described reception process of the telemetry data TM and the above-mentioned reception process of the image signal are performed asynchronously.

[0057] <4. Image Synthesis Processing as an embodiment> Here, in a system that displays images captured by a camera 82 mounted on an unmanned moving body such as a drone 2 to a user such as a pilot, if the captured images are transmitted in a compressed state to the control device 1 side in a situation where the radio wave reception condition gradually deteriorates, the control device 1 side will at some point be unable to decode the images, and the displayed images will suddenly cease to be displayed. As described above, such a situation may hinder the stable movement of the unmanned moving body.

[0058] Therefore, in this embodiment, as described above, the aerial station 80 transmits images captured by the camera 82 in an uncompressed state. Then, on the pilot device 1 side, the image synthesis unit 58 of the ground station 50 performs the following image synthesis process on the image signals received by the multiple first communication antennas (dipole antennas 21, 22, patch antennas 24, 25). In other words, this is an image synthesis process in which, for image signals received by multiple first communication antennas, one image signal is selected based on an evaluation value of the reception state for each block consisting of a predetermined number of pixels, and the selected image signal is mapped within an image frame for one frame.

[0059] An example of a configuration for implementing the image synthesis processing according to the embodiment will be described with reference to the block diagram of FIG. Fig. 11 shows the internal configuration of the ground station 50, mainly related to the synthesis of image signals received by a plurality of antennas, together with the dipole antennas 21 and 22 and the patch antennas 24 and 25 shown in Fig. 10. Fig. 11 also shows an example of the internal configuration of each of the radio wave selection units 56 and 57.

[0060] As shown in the figure, the radio wave selection unit 56 has demodulation units 90-1 and 90-4, error rate calculation units 91-1 and 91-4, and a selection unit 92. The radio wave selection unit 57 has demodulation units 90-5 and 90-2, error rate calculation units 91-1 and 91-2, and a selection unit 93.

[0061] In the radio wave selection unit 56, the demodulation unit 90-1 demodulates the signal received by the receiving circuit 51, and the demodulation unit 90-4 demodulates the signal received by the receiving circuit 54. The error rate calculation unit 91-1 calculates the error rate for the image signal received via the dipole antenna 21, based on the error detection code obtained by the demodulation process of the demodulation unit 90-1. Moreover, the error rate calculation unit 91-4 calculates the error rate for the image signal received via the patch antenna 24, based on the error detection code obtained by the demodulation process by the demodulation unit 90-4.

[0062] In the radio wave selection unit 57, the demodulation unit 90-5 demodulates the signal received by the reception circuit 55, and the demodulation unit 90-2 demodulates the signal received by the reception circuit 52. The error rate calculation unit 91-5 calculates the error rate for the image signal received via the patch antenna 25 based on the error detection code obtained by the demodulation processing by the demodulation unit 90-5, and the error rate calculation unit 91-2 calculates the error rate for the image signal received via the dipole antenna 22 based on the error detection code obtained by the demodulation processing by the demodulation unit 90-2.

[0063] Here, for the error rate calculation in the error rate calculation units 91-1, 91-4, 91-5, and 91-2, the minimum data unit for which the error rate calculation can be performed is defined. As an example for explanatory purposes, in this example, it is assumed that this minimum data unit is defined as a unit of 8×8=64 pixels. The error rate calculation here is based on the results of error detection using error detection codes such as parity bits contained in the transmitted data.

[0064] In the radio wave selection unit 56, the selection unit 92 inputs block-unit image signals obtained by demodulation processing from each of the demodulation units 90-1, 90-4, and selects one of the input block-unit image signals based on the error rates of each image signal calculated by the error rate calculation units 91-1, 91-4. Here, the block unit is a unit larger than the minimum data unit for error rate calculation, and specifically, the block unit is an integer multiple of a plurality of the minimum data units. For example, in this example, the minimum data unit is a unit of 8×8=64 pixels, while the block unit is a unit of 32×32=1024 pixels. That is, in this example, one block of the image contains 4×4=16 pixel groups equivalent to the minimum data unit. Hereinafter, the pixel group of the minimum data unit in error rate calculation will be referred to as the "minimum data unit portion."

[0065] The selection unit 92 calculates an error rate for the block-unit image signals input from the demodulation units 90-1 and 90-4, i.e., an error rate for each block, based on the error rate for each minimum data unit input from the error rate calculation units 91-1 and 91-4. The error rate for each block is calculated, for example, as an average value of the error rates for the minimum data units. Then, the selection unit 92 selects one of the block-unit image signals based on the error rate calculated for the block-unit image signals input from the demodulation units 90-1 and 90-4. Specifically, the selection unit 92 selects the image signal with the smaller calculated error rate. The selection unit 92 outputs the selected block-unit image signal and the error rate calculated for the selected block-unit image signal to the image synthesis unit 58.

[0066] In the radio wave selection unit 57, the selection unit 93 inputs block-unit image signals obtained by demodulation processing from each of the demodulation units 90-5 and 90-2, and selects one of the input block-unit image signals based on the error rates of each image signal calculated by the error rate calculation units 91-5 and 91-2. Specifically, the selection unit 93 calculates an error rate (error rate in units of blocks) for the image signals in units of blocks input from the demodulation units 90-5, 90-2 based on the error rate for each minimum data unit input from the error rate calculation units 91-5, 91-2, and selects one of the image signals in units of blocks input from the demodulation units 90-5, 90-2 based on the error rate. Specifically, the selection unit 93 selects the image signal with the smaller calculated error rate. The selection unit 93 outputs the selected block-unit image signal and the error rate calculated for the selected block-unit image signal to the image synthesis unit 58.

[0067] In addition, in the radio wave selection unit 56, the block-unit image signals from each antenna (21, 24) obtained by the demodulation units 90-1, 90-4, respectively, and the error rates for each of the image signals calculated by the error rate calculation units 91-1, 91-4 are temporarily stored in the memory unit 62. In addition, in the radio wave selection unit 57, the block-unit image signals from each antenna (25, 22) obtained by the demodulation units 90-5, 90-2, respectively, and the error rates for each of these image signals calculated by the error rate calculation units 91-5, 91-2 are temporarily stored in the memory unit 62.

[0068] Here, the image synthesis process by the image synthesis unit 58 is performed for each frame of the image signal as described above. Correspondingly, the image signal and the error rate are stored in the memory unit 62 in units of blocks as described above for each frame.

[0069] FIG. 12 is a functional block diagram for explaining the functions of the image synthesis unit 58. As shown in FIG. As shown in the figure, the image synthesis unit 58 has functions as an image mapping unit F1, a missing portion reselection unit F2, a missing portion interpolation processing unit F3, and a high resolution processing unit F4.

[0070] The image mapping unit F1 represents the function of performing the image synthesis process described above. Specifically, the image mapping unit F1 selects the image signal with the lower error rate from the image signals in units of blocks input from the selection units 92 and 93 based on the error rates of the image signals input from the selection units 92 and 93, and maps the image signal within an image frame for one frame. The image generated by this mapping is hereinafter referred to as a "mapping image."

[0071] By carrying out the image synthesis process as described above for image signals transmitted in an uncompressed state, it is possible to continue displaying images for at least some of the blocks, unless a reception error occurs for the image signals of all blocks of the image. In other words, it is possible to avoid the occurrence of a situation in which image display suddenly stops due to deterioration of the reception state, as occurs in the case of decoding and displaying compressed images. Therefore, the continuity of image display can be improved even when the reception conditions deteriorate, and the movement stability of the unmanned moving body can be improved.

[0072] The missing portion reselection unit F2 selects one image signal from the image signals received by the multiple first communication antennas based on an evaluation value of the reception state for a missing portion that is a reception error portion that occurs in the mapping image by the image mapping unit F1, in units of small regions Sm consisting of a smaller number of pixels than the number of pixels in a block unit. A missing part as a reception error part means a part (image part) where data could not be demodulated due to a deterioration in the reception state, specifically, for example, a part where error correction processing using an error correction code could not be performed in the demodulation process. In this case, the unit (image range) where error correction is performed using the error correction code is the unit where a missing part occurs. The missing portion reselection unit F2 performs a process of selecting image signals with good reception conditions from the received image signals of each antenna for such missing portions in the mapping image, in units of small regions Sm defined as areas with a smaller number of pixels than the number of pixels in a block unit.

[0073] FIG. 13 illustrates the relationship between blocks Bl, which are block-unit regions, and small regions Sm. "Px" in the figure refers to each pixel that makes up the image (hereinafter, the pixels that make up the image will be referred to as "pixel Px"). The small region Sm is defined as a region made up of multiple pixels Px, and more specifically, as a region of 2 x 2 = 4 pixels or more. As an example for the purpose of explanation, it is assumed here that the small region Sm is a region of 8 x 8 = 64 pixels. A block Bl is defined as an area including a plurality of small areas Sm, specifically, includes 2×2=4 or more small areas Sm. As an example for the purpose of explanation, it is assumed here that a block Bl includes 4×4=16 small areas Sm (1024 pixels).

[0074] Fig. 14 shows an example of a mapping image in which a missing portion has occurred. The blackened areas indicate the missing portions. Here, an example is shown in which missing parts occur in units of small regions Sm. This is because the size of the error correction frame as a unit for performing error correction is the same as the size of the small region Sm. Note that matching the size of the small region Sm with the size of the error correction frame is merely an example, and the size of the small region Sm may be larger than the size of the error correction frame.

[0075] The missing portion reselection unit F2 selects an image signal for each small region Sm for the missing portion based on the error rate stored in the memory unit 62. As described above, the memory unit 62 stores the error rates (error rates for each antenna) calculated by the error rate calculation units (91-1, 91-4, 91-5, 91-2) for the frame to be processed, and the image signals obtained by the demodulation units (90-1, 90-4, 90-5, 90-2). The missing portion reselection unit F2 evaluates the reception state of each antenna for the target small region Sm based on the error rate for each image signal of each antenna (21, 24, 25, 22) stored in the memory unit 62. At this time, if the size of the small region Sm is larger than the size of the minimum data unit for the error rate calculation described above, the error rate for each small region Sm is calculated (for example, the average value, etc.) using the error rate of each minimum data unit part included in the small region Sm. As mentioned above, in this example, the size of the minimum data unit portion is the same as the size of the small region Sm, which is 8 x 8 = 64 pixels, so the error rate of the corresponding minimum unit data portion can be used as is for the error rate of the small region Sm. Then, the missing portion reselection unit F2 selects one image signal as the image signal of the target small region Sm based on the error rate for each antenna for the small region Sm. Specifically, it selects the image signal with the smallest error rate.

[0076] The missing part reselection unit F2 selects an image signal for each small region Sm for each missing part as described above, and maps the selected image signal. That is, the image signal of the missing part is updated to the selected image signal in the mapping image.

[0077] By the processing of the missing part reselection unit F2 as described above, if the image signal received by another antenna has a better reception condition for a missing part that occurs in mapping based on the reception condition evaluation on a block-by-block basis, the missing part can be interpolated using the image signal received by that antenna.

[0078] 12, the missing portion interpolation processor F3 uses an image filter to interpolate missing portions occurring in the mapping image. Specifically, the missing portion interpolation processor F3 in this example performs an interpolation process using an image filter on missing portions that continue to occur in the mapping image after the missing portion reselection unit F2 has performed the interpolation. The interpolation process here may be a linear interpolation process using values ​​of multiple pixels Px surrounding the target pixel Px (pixel Px of the missing portion).

[0079] By performing such interpolation processing by the missing portion interpolation processing unit F3, it is possible to further reduce the amount of missing image information due to deterioration of the reception conditions, and to further improve the continuity of image display even when the reception conditions deteriorate.

[0080] It is also possible to configure the mapping image obtained by the image mapping unit F1 to be subjected to an interpolation process by the missing portion interpolation processing unit F3 without performing the process by the missing portion reselection unit F2.

[0081] The high-resolution processing unit F4 performs high-resolution processing on the mapping image, which includes upscaling processing to increase the resolution and processing to correct noise such as jaggies generated by the upscaling processing. The resolution enhancement processor F4 of this example performs resolution enhancement processing on the mapping image after the interpolation processing by the missing portion interpolation processor F3.

[0082] The high resolution process may be performed on the mapping image obtained by the image mapping unit F1 without performing the interpolation process by the missing portion reselection unit F2 or the missing portion reselection unit F2 or the interpolation process by the missing portion interpolation unit F3.

[0083] 15 is a flowchart showing an embodiment of processing performed by the image synthesis unit 58. Specifically, this is a flowchart of processing corresponding to the above-mentioned image mapping unit F1, missing portion reselection unit F2, missing portion interpolation processing unit F3, and high resolution processing unit F4. The image synthesis unit 58 repeatedly executes the process shown in FIG. 15 for each frame of the captured image.

[0084] First, in step S101, the image synthesis unit 58 selects the block unit image with the smallest error rate and maps it within the image frame for one frame. That is, for the image signals input from the radio wave selection units 56 and 57, the image signal with the smallest error rate (the smaller one in this example) is selected on a block basis and mapped within the image frame for one frame.

[0085] In step S102 following step S101, the image synthesis unit 58 determines whether or not there is a missing portion in the mapping image generated in step S101. If there is no missing portion in step S102, the image synthesis unit 58 proceeds to step S107, where it performs the above-mentioned high resolution processing on the mapping image, and ends the series of processing steps shown in FIG.

[0086] On the other hand, if there is a missing portion in step S102, the image synthesis unit 58 proceeds to step S103, where it selects an image signal with the minimum error rate from the image signals received by all antennas for the missing portion of the mapping image in units of small regions Sm, and then performs processing to map the selected image signal in the following step S104. The processing in steps S103 and S104 is similar to that explained as the processing in the missing portion reselection unit F2 described above, and therefore a duplicate explanation will be avoided.

[0087] In step S105 following step S104, the image synthesis unit 58 determines whether or not there is a missing portion. That is, the image synthesis unit 58 determines whether or not there is a missing portion in the mapping image that has been subjected to the mapping process in step S104.

[0088] If there is no missing portion in step S105, the image synthesis unit 58 proceeds to step S107, where it performs the above-mentioned high resolution processing on the mapping image, and ends the series of processing steps shown in FIG.

[0089] On the other hand, if there is a missing portion in step S105, the image synthesis unit 58 performs an interpolation process in step S106. This interpolation process is similar to the process of the missing portion interpolation processor F3 described above, and therefore a duplicated description will be avoided.

[0090] In step S107 following step S106, the image synthesis unit 58 performs a high-resolution process. If the interpolation process in step S106 has been performed, the high-resolution process is performed on the mapping image after the interpolation process.

[0091] In response to executing the process of step S107, the image synthesis unit 58 ends the series of processes shown in FIG.

[0092] Fig. 16 and Fig. 17 are diagrams for explaining the effect of the image synthesis processing as an embodiment, Fig. 16 is an explanatory diagram of a state where the radio wave reception state is relatively good, and Fig. 17 is an explanatory diagram of a state where the radio wave reception state is deteriorated. In each diagram, Fig. A shows an image of a display image obtained when the image synthesis processing as an embodiment is performed, and Fig. B shows an image of a display image obtained in a conventional method of transmitting compressed images for comparison.

[0093] When the reception conditions in FIG. 16 are relatively good, in the conventional method (FIG. 16B), the decoding process functions properly, so that an image without any loss can be obtained even if a reception error occurs in part of the image. On the other hand, in the image synthesis process of the embodiment, if a reception error occurs in part of the image, the displayed image is likely to be missing. Therefore, even if the reception condition is relatively good, the displayed image is more likely to be missing than in the case of FIG. 16B.

[0094] As described above, in the conventional method, when the reception conditions deteriorate, the image display is interrupted at a certain point; specifically, an error image such as an all-black image as shown in FIG. 17B is displayed. In contrast, according to the image synthesis process of the embodiment, although noise (missing parts) in the displayed image increases as the reception condition deteriorates, the successfully received parts can continue to be displayed (FIG. 17A). In other words, as described above, it is possible to improve the continuity of image display even when the reception condition deteriorates.

[0095] <5. Modifications> The embodiment is not limited to the specific example described above, and various modified configurations may be adopted. For example, in the above, a drone is used as an example of an unmanned mobile body, but in the present invention, the unmanned mobile body is not limited to the form of an unmanned aerial vehicle such as a drone, and can also take the form of an unmanned ground vehicle, an unmanned surface ship, an unmanned surface boat, an unmanned submarine boat, etc.

[0096] In the above, the reception state is evaluated using the error rate as an evaluation value, but the reception state may also be evaluated based on other evaluation indices such as radio wave intensity.

[0097] In the above, an example was given in which the type of antenna to be input to each of radio wave selection units 56 and 57, which selects one of the image signals input from a plurality of antennas, is different between a dipole antenna and a patch antenna. However, it is also possible to configure each of radio wave selection units 56 and 57 to input image signals from multiple antennas of the same type, such as inputting image signals from multiple dipole antennas (e.g., dipole antennas 21 and 22) to radio wave selection unit 56 and inputting image signals from multiple patch antennas (e.g., patch antennas 24 and 25) to radio wave selection unit 57.

[0098] In the embodiment, the patch antennas 24 and 25 and the dipole antennas 21 and 22 are used as the first communication antennas, but the antenna configuration is not limited to this. Other types of antennas may be used, and the number of antennas may be increased or decreased.

[0099] <6. Summary of the embodiment> As described above, the communication system as an embodiment is a communication system between an unmanned mobile body (drone 2) and a control device (same as 1), and simultaneously executes a first wireless communication in which an image signal captured by a camera (same as 82) attached to the unmanned mobile body is transmitted to the control device by one-way communication in a first frequency band, and a second wireless communication in which a control signal is transmitted from the control device to the unmanned mobile body and telemetry data is transmitted from the unmanned mobile body to the control device by simplex communication in a second frequency band lower than the first frequency band. In the first wireless communication, the image signal is transmitted to the control device in an uncompressed state, and the control device receives the image signals by multiple receiving antennas and selects one image signal from the image signals received by the multiple receiving antennas based on an evaluation value of the reception state in block units consisting of a predetermined number of pixels, and maps the selected image signal within an image frame for one frame. As described above, the image signal is transmitted in an uncompressed state, the uncompressed image signal is received by a plurality of receiving antennas, and one image signal is selected from the image signals received by the antennas based on an evaluation of the reception state for each predetermined block, and the selected image signal for each block is mapped to one image. As long as there is no reception error for the image signals of all blocks of the image, it is possible to continue displaying the image for at least a portion of the blocks. In other words, it is possible to avoid the occurrence of a situation in which the image display suddenly stops due to the deterioration of the reception state, as in the case of decoding and displaying a compressed image. Therefore, it is possible to improve the continuity of image display even when reception conditions deteriorate, and a system can be realized that can improve the movement stability of the unmanned mobile body in a communication system between the unmanned mobile body and a control device.

[0100] Furthermore, when transmitting image signals wirelessly in an uncompressed state, there is a concern that the communication bandwidth may be constrained. However, necessary signals other than images (control signals and telemetry data) are transmitted using a second frequency band different from the communication frequency band for image signals (first frequency band), so stable transmission can be achieved for both the transmission of image signals and the transmission of control signals and telemetry data. Furthermore, since the image signal is transmitted by one-way communication, the antenna used to receive the image signal can be a reception-only antenna, which allows the reception gain (antenna gain) to be set high, thereby improving the reception sensitivity of the image signal from a long distance. This makes it possible to stabilize image transmission even over long distances, and is particularly suitable for transmitting large amounts of data as uncompressed images. For clarity, a transmitting / receiving antenna that transmits and receives is subject to antenna gain restrictions on the transmitting side. If it is for reception only, there is no such antenna gain restriction, and the antenna gain can be increased.

[0101] Furthermore, in the communication system of this embodiment, since there is no need to compress / decompress images, the processing time required from capturing an image to displaying it can be shortened, and delays in image display can also be reduced.

[0102] A control device (1) according to an embodiment of the present invention simultaneously executes a first wireless communication, which is a one-way communication in a first frequency band, and a second wireless communication, which is a simplex communication in a second frequency band lower than the first frequency band, with an unmanned mobile body, receives an image signal captured by a camera attached to the unmanned mobile body and transmitted in an uncompressed state by a transmitting device (aerial station 80) mounted on the unmanned mobile body through the first wireless communication, and transmits a control signal to the unmanned mobile body and receives telemetry data from the unmanned mobile body through the second wireless communication. The ground station is equipped with a plurality of first communication antennas (dipole antennas 21, 22 and patch antennas 24, 25) used for receiving wireless communications, and a second communication antenna (dipole antenna 23) used for transmitting and receiving second wireless communications, and has an image mapping unit (F1) that selects one image signal from the image signals received by the plurality of first communication antennas, for each block consisting of a predetermined number of pixels, based on an evaluation value of the reception state, and maps the selected image signal within an image frame for one frame. This makes it possible to continue displaying images for at least some of the blocks as long as there are no reception errors in the image signals for all blocks of the image, which makes it possible to avoid the occurrence of a situation in which image display suddenly stops due to deterioration of the reception conditions, as occurs when a compressed image is decoded and displayed. Therefore, it is possible to improve the continuity of image display even when reception conditions deteriorate, and a system can be realized that can improve the movement stability of the unmanned mobile body in a communication system between the unmanned mobile body and a control device.

[0103] In addition, in the control device as an embodiment, the ground station has a missing part reselection unit (F2) that selects one image signal from the image signals received by the multiple first communication antennas based on an evaluation value of the reception state, in units of small areas consisting of a smaller number of pixels than the number of pixels in a block unit, for missing parts that are reception error parts that occur in the mapping image by the image mapping unit. Even if the result of the reception state evaluation on a block-by-block basis for image signals received by multiple antennas is the best, there is no guarantee that the reception state of each small area in that block is better than that of other antennas, and for a certain small area, the reception state of the image signal received by another antenna may be better. Therefore, for missing parts in the mapping image as described above, the reception state evaluation of the image signals received by multiple antennas is performed on a small-area basis, and one image signal is selected. As a result, for missing parts generated by mapping based on the reception state evaluation on a block-by-block basis, if the reception state of the image signal received by another antenna is better, the missing parts can be interpolated with the image signal received by that antenna. Therefore, it is possible to further reduce the amount of image information lost due to deterioration of reception conditions, and further improve the continuity of image display due to deterioration of reception conditions, thereby further improving the movement stability of unmanned moving bodies.

[0104] Furthermore, in the control device according to the embodiment, the ground station has a missing portion interpolation processing unit (F3) that uses an image filter to interpolate missing portions that occur in the mapping image generated by the image mapping unit. This makes it possible to further reduce the amount of image information lost due to deterioration of reception conditions, and further improve the continuity of image display even when reception conditions deteriorate, thereby further improving the movement stability of unmanned mobile bodies.

[0105] Furthermore, in the embodiment of the pilot control device, the multiple first communication antennas include patch antennas. In order to transmit images from long distances, multiple patch antennas (24, 25) are provided as the first communication antenna. The patch antennas have a relatively narrow directivity and high gain, making them suitable for long-distance communication.

[0106] In the pilot control device as an embodiment, the multiple first communication antennas include a patch antenna and a dipole antenna. The patch antenna allows image transmission from long distances, while the dipole antenna allows for wide-directivity reception of image signals when an unmanned mobile vehicle is flying close by. Therefore, appropriate image reception can be achieved regardless of the flying position of the unmanned mobile body.

[0107] Furthermore, in the control device as an embodiment, the multiple first communication antennas include patch antennas that are movable so that the orientation of the receiving surface can be changed. By making the relatively narrow-directivity patch antenna a movable antenna that can change the direction of the receiving surface, the user can adjust the reception condition. Specifically, the user can manually adjust the image quality to the best possible quality while viewing the captured image on a display device such as a tablet PC 3, making the adjustment easy and making it easy to obtain good quality captured images. In other words, because this is a system in which the user controls the unmanned mobile body while viewing the captured image at hand, it can be said that the user can easily adjust the reception condition by adjusting the movement of the patch antenna. [Explanation of symbols]

[0108] 1. Controls 2. Drone 3. Tablet PC 10 Main body 21, 22, 23 Dipole antenna 24,25 Patch antenna 50 Ground Station 51, 52, 54, 55 Receiver circuit 53 Transmitting and receiving circuit 56,57 Radio wave selection unit 58 Image synthesis unit 59 Image Settings 60 CPU 61 Modulation and demodulation section 62 Memory section 80 Aerial station 82 Camera 83 Flight Controller 84,85 Antenna 90-1, 90-2, 90-4, 90-5 Demodulation section 91-1, 91-2, 91-4, 91-5 Error rate calculation section 92,93 Selection section Px Pixels Sm small area Bl Block F1 Image Mapping Section F2 Reselect missing part F3 Missing area interpolation processing section F4 High-resolution processing section

Claims

1. A communication system between an unmanned vehicle and a control device, comprising: A first wireless communication that transmits an image signal captured by a camera attached to the unmanned moving body to the control device by one-way communication in a first frequency band; A second wireless communication is simultaneously performed by transmitting a control signal from the control device to the unmanned vehicle and transmitting telemetry data from the unmanned vehicle to the control device by simplex communication in a second frequency band lower than the first frequency band, and In the first wireless communication, transmitting the image signal in an uncompressed form to the flight control device; In the control device, receiving the image signals by a plurality of receiving antennas; Among the image signals received by the plurality of receiving antennas, one image signal is selected based on an evaluation value of a receiving state for each block consisting of a predetermined number of pixels, and the selected image signal is mapped within an image frame for one frame. Communication systems.

2. a ground station which simultaneously executes a first wireless communication, which is a one-way communication in a first frequency band, and a second wireless communication, which is a simplex communication in a second frequency band lower than the first frequency band, with an unmanned mobile body, receives an image signal captured by a camera attached to the unmanned mobile body and transmitted in an uncompressed state by a transmitting device mounted on the unmanned mobile body, by the first wireless communication, and transmits a control signal to the unmanned mobile body and receives telemetry data from the unmanned mobile body by the second wireless communication; a plurality of first communication antennas used for receiving the first wireless communication; a second communication antenna used for transmitting and receiving the second wireless communication, The ground station: an image mapping unit that selects one image signal based on an evaluation value of a reception state for each block of a predetermined number of pixels from the image signals received by the plurality of first communication antennas, and maps the selected image signal within an image frame for one frame; Control device.

3. The ground station: a missing portion reselection unit that selects one image signal based on an evaluation value of a reception state from the image signals received by the plurality of first communication antennas in units of small regions each having a smaller number of pixels than the number of pixels in the block unit, for a missing portion as a reception error portion occurring in the mapping image by the image mapping unit; The control device according to claim 2.

4. The ground station: a missing portion interpolation processing unit that uses an image filter to interpolate a missing portion generated in the mapping image by the image mapping unit; The control device according to claim 2.

5. The plurality of first communication antennas include a patch antenna. A control device according to any one of claims 2 to 4.

6. The plurality of first communication antennas include a patch antenna and a dipole antenna. A control device according to any one of claims 2 to 4.

7. The plurality of first communication antennas include a patch antenna that is movable so that the orientation of the receiving surface can be changed. A control device according to any one of claims 2 to 4.

Images