Control device and information processing system

Abstract

A control device (100) incorporated in a moving body (10) comprises: an image recognition unit (13) that performs an image recognition process on image data indicating an image captured by an imaging device mounted on the moving body; a first control unit (14) that moves the imaging device in a direction including at least a vertical-direction component in accordance with the result of the image recognition process; and a second control unit (15) that controls the movement of the moving body and stops the moving body at least at the imaging timing of the imaging device.

Description

Control Device and Information Processing System

[0001] The present invention relates to a technique for imaging an image by an imaging device mounted on a moving body.

[0002] Gaussian Splatting is known as one of the techniques for generating 3D data using 2D data captured from multiple viewpoints. For example, in Patent Document 1, when a self-propelled inspection robot stops at an inspection target imaging location and the camera is directed at the inspection target, a mechanism for evaluating the imaging difficulty based on the amount of 3D point cloud in front of the inspection target among the 3D point clouds within the imaging angle is disclosed.

[0003] Japanese Patent Application Laid-Open No. 2023-72514

[0004] When generating 3D image data using Gaussian Splatting, if the position of the imaging device in the height direction is fixed, only image data from the same viewpoint in the height direction can be obtained, and for example, the accuracy when generating 3D image data for objects with different shapes in the height direction may decrease.

[0005] Therefore, an object of the present invention is to improve the accuracy when generating 3D image data using image data indicating an image captured by an imaging device mounted on a moving body.

[0006] To solve the above problems, the present invention provides an image recognition unit that performs image recognition processing on image data indicating an image captured by an imaging device mounted on a moving body, a first control unit that moves the imaging device in a direction including at least a vertical direction component according to the result of the image recognition processing, and a second control unit that controls the movement of the moving body, the second control unit stopping the moving body at least at the timing of imaging by the imaging device.

[0007] Further, the present invention provides an information processing system including the control device and an information processing device that generates 3D image data using 2D image data indicating images captured from different viewpoints by the imaging device.

[0008] According to the present invention, it is possible to improve the accuracy when generating three-dimensional image data using image data representing an image captured by an imaging device mounted on a moving object.

[0009] This figure shows an example of the appearance of a mobile body 10 according to one embodiment of the present invention. This is a block diagram showing an example of the electrical configuration of the mobile body 10 according to the same embodiment. This is a block diagram showing an example of the functional configuration of the control device 10 provided in the mobile body 10 according to the same embodiment. This figure illustrates the relationship between the mobile body 10 and object O according to the same embodiment. This figure illustrates the relationship between the mobile body 10 and object O according to the same embodiment. This figure illustrates the relationship between the mobile body 10 and object O according to the same embodiment. This is a flowchart illustrating the operation of the mobile body 10 according to the same embodiment.

[0010] [composition]

[0011] A mobile body 10 according to one embodiment of the present invention moves along a horizontal road surface R and takes images of its surroundings. This obtains two-dimensional image data of each object around the mobile body 10 from multiple viewpoints (positions where imaging is performed), and three-dimensional data (3D Gaussian) is generated using Gaussian splatting with this two-dimensional data. From this three-dimensional data, a two-dimensional image viewed from any viewpoint can be reconstructed. In this embodiment, an example using Gaussian splatting is described, but in the present invention, the technique for generating three-dimensional data using two-dimensional data taken from multiple viewpoints is not limited to Gaussian splatting.

[0012] As shown in Figure 1, the mobile body 10 comprises a main body 110, a plurality of wheels 120, an imaging device 130, and a support member 140 that supports the imaging device 130. The support member 140 includes a slide control mechanism consisting of a member extending in the height direction (vertical direction), which allows the imaging device 130 to slide vertically relative to the main body 110. In other words, the imaging device 130 can capture images of each object around the mobile body 10 from different horizontal viewpoints as the mobile body 10 moves horizontally, and can also capture images of each object around the mobile body 10 from different vertical viewpoints as the support member 140 moves the imaging device 130 vertically.

[0013] Figure 2 shows an example of the electrical configuration of the mobile body 10. Physically, the mobile body 10 is configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, user interface device 1005, imaging device 130, vertical sliding mechanism 1007, wheel control mechanism 1008, and a bus connecting these. In the following description, the word "device" can be read as a circuit, device, unit, etc. The electrical configuration of the mobile body 10 may be configured to include one or more of the devices shown in the figure, or it may be configured to omit some of the devices.

[0014] Each function in the mobile device 10 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, which allows the processor 1001 to perform calculations, control communication by the communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and storage 1003.

[0015] The processor 1001 controls the entire computer, for example, by running an operating system. The processor 1001 may consist of a central processing unit (CPU) that includes interfaces with peripheral devices, control units, arithmetic units, registers, and so on.

[0016] The processor 1001 reads programs (program code), software modules, data, etc., from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes accordingly. The program used is one that causes a computer to execute at least a part of the operations described later. Functional blocks of the mobile body 10 may be stored in the memory 1002 and implemented by control programs that run on the processor 1001. Various processes may be executed by one processor 1001, but may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may also be transmitted to the mobile body 10 via a telecommunications line.

[0017] The memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. The memory 1002 may also be called a register, cache, main memory, etc. The memory 1002 can store executable programs (program code), software modules, etc., for carrying out the method according to this embodiment.

[0018] The storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device.

[0019] The communication device 1004 is hardware (transmitting / receiving device) for communicating between computers via a communication network, and is also called a network device, network controller, network card, or communication module. The communication device 1004 communicates data with the information processing device 200 via a communication network (not shown). The information processing device 200 generates three-dimensional image data using a technique called Gaussian Splatting, which is based on two-dimensional image data showing images captured from multiple different viewpoints by the imaging device 130. The information processing system is constructed by the control device 100 (described later) provided by the mobile body 10 and the information processing device 200, which generates three-dimensional image data using two-dimensional image data showing images captured from multiple viewpoints.

[0020] Each device, such as the processor 1001 and the memory 1002, is connected by a bus for communicating information. The bus may be configured using a single bus, or different buses may be configured for each device. The control device 100 according to this embodiment is realized by a computer consisting of the processor 1001, the memory 1002, and the storage 1003.

[0021] The user interface device 1005 includes an input device (e.g., keys, microphone, switch, button, etc.) that receives input from the user and an output device (e.g., display, speaker, LED lamp, etc.) that provides output to the user. The input device and the output device may be configured as an integrated unit (e.g., a touchscreen).

[0022] The imaging device 130 is a device that generates two-dimensional image data representing the captured image, and is, for example, a 180-degree camera.

[0023] The vertical sliding mechanism 1007 is a mechanism that moves the imaging device 130 vertically relative to the main body 110. The vertical sliding mechanism 1007 is a mechanism that allows the imaging device 130 to move along a support member that extends vertically.

[0024] The wheel control mechanism 1008 is a mechanism for controlling the wheel 120 and includes a motor mechanism for rotating the wheel 120 and a steering mechanism for changing the orientation of the wheel 120's axle.

[0025] The mobile unit 10 may also be composed of hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by this hardware. For example, the processor 1001 may be implemented using at least one of these hardware components.

[0026] Figure 3 is a block diagram showing an example of the functional configuration of the mobile device 10. In the mobile device 10, the hardware components shown in Figure 2 work together to realize the functions of an acquisition unit 11, a storage unit 12, an image recognition unit 13, a first control unit 14, and a second control unit 15, as shown in Figure 3.

[0027] The acquisition unit 11 acquires image data representing images captured by the imaging device 130. This image data may represent a still image or a moving image. The acquired image data is stored in the storage unit 12.

[0028] The image recognition unit 13 performs image recognition processing on the image data acquired by the acquisition unit 11. The storage unit 12 stores an object database (DB), which is a data set representing the shapes (patterns) of various objects (for example, desks, chairs, walls, pillars, potted plants, personal computers, etc., in an office). The image recognition unit 13 uses a technique called pattern matching to determine whether there are any images in the image data acquired by the acquisition unit 11 that have a similarity to the shapes (patterns) of various objects included in the object DB within a threshold, thereby recognizing images corresponding to objects in the image data. For each shape (pattern) of an object included in the object DB, a flag is pre-set to indicate whether or not it is an object that differs in shape or size in the vertical direction.

[0029] The first control unit 14 controls the vertical sliding mechanism 1007 to move the imaging device 130 up and down according to the result of the image recognition processing by the image recognition unit 13. Specifically, if the image recognition processing does not recognize image data corresponding to objects with different shapes or sizes in the vertical direction within a certain threshold distance from the moving body 10, the first control unit 14 does not move the imaging device 130 up and down. For example, if only images corresponding to walls, pillars, etc., are recognized, the imaging device 130 will not move up and down, and these objects will be imaged from the same viewpoint in the vertical direction. For example, as shown in Figure 4, when the imaging device 130 images an object O which is a wall, this object O has the same shape or size in the vertical direction, and imaging from the same viewpoint in the vertical direction is sufficient to reproduce a three-dimensional image.

[0030] On the other hand, the first control unit 14 moves the imaging device 130 up and down if, through image recognition processing, image data corresponding to an object with a different shape or size in the vertical direction is recognized within a certain threshold distance range from the moving object 10. For example, if images corresponding to a desk, chair, potted plant, personal computer, etc., are recognized, the imaging device 130 is moved up and down, and images are taken of these objects from different viewpoints in the vertical direction. As shown in Figures 5 to 7, for example, an object O, which is a desk, has different shapes or sizes in the vertical direction. Therefore, if images are taken only from a certain viewpoint in the vertical direction, the shape or size of the object above or below that viewpoint will not be captured with sufficient accuracy. Therefore, for such objects, images are taken from different viewpoints in the vertical direction to accurately reproduce the shape and size of the object in three dimensions.

[0031] Returning to the explanation of Figure 4, the second control unit 15 controls the movement of the mobile body 10. The path that the mobile body 10 takes as it moves while imaging in the space to be imaged is predetermined, and path data indicating that path is stored in the storage unit 12. The second control unit 15 refers to this path data and controls the wheels 120 so that the mobile body 10 moves along that path. The imaging device 130 takes images of the area around the mobile body 10, for example, each time the mobile body 10 moves a predetermined distance or each time a predetermined amount of time has elapsed.

[0032] For example, the second control unit 15 stops the moving body 10 at least during the period when the imaging device 130 is capturing an image, while it is controlling the movement of the moving body 10. This is because capturing an image of an object with the moving body 10 stopped results in less image blur compared to capturing an image of an object while the moving body 10 is moving.

[0033] Furthermore, if the second control unit 13 recognizes image data corresponding to an object with a different shape or size in the vertical direction through image recognition processing, the second control unit 15 moves the mobile body 10 away from the path indicated by the path data and in a direction that approaches that object (see Figures 5-7). This is because imaging with the mobile body 10 closer to these objects allows for the capture of more detailed images compared to imaging with the mobile body 10 not close to these objects. Once imaging is completed with the mobile body 10 moving in the direction that approaches the object, the second control unit 15 returns to the path indicated by the path data and controls the wheels 120 so that the mobile body 10 moves along the path.

[0034] [Operation] Next, the operation of the mobile unit 10 will be described. The procedures for each process shown in Figure 8 are described in the program stored in the mobile unit 10.

[0035] In Figure 8, once image data is acquired by the acquisition unit 11 (step S11), the image recognition unit 13 performs image recognition processing on the image data acquired by the acquisition unit 11 (step S12).

[0036] If, as a result of the image recognition processing by the image recognition unit 13, image data corresponding to an object with a different shape or size in the vertical direction is recognized within a certain threshold distance range from the moving body 10 (step S13; YES), the second control unit 15 controls the wheels 120 to move the moving body 10 toward the object (step S14). Then, the first control unit 14 controls the vertical sliding mechanism 1007 to move the imaging device 130 up and down. The imaging device 130 then images the object O (step S16). Once the number of images corresponding to the difference in shape or size in the vertical direction has been taken (step S17; YES), the process returns to step S11.

[0037] According to the embodiments described above, for objects with different shapes or sizes in the vertical direction, it is possible to improve the accuracy of generating three-dimensional image data by acquiring image data captured from multiple different viewpoints in the vertical direction. On the other hand, for objects such as walls and columns, where changing the viewpoint in the vertical direction does not make sense, imaging is performed from the same viewpoint, thus reducing the time required to image the space being imaged.

[0038] [Modifications] The present invention is not limited to the embodiments described above. The embodiments described above may be modified as follows. Furthermore, two or more of the following modifications may be combined and implemented.

[0039] [Modification 1] When the first control unit 14 recognizes image data corresponding to an object with a different shape or size in the vertical direction through image recognition processing, it may vary the distance or number of times it moves the imaging device 130 in a direction that includes at least a vertical component, according to the shape or size of the object. For example, the first control unit 14 may stop the vertical movement of the imaging device 130 and perform imaging each time the shape of the object differs in the vertical direction, or stop the vertical movement of the imaging device 130 and perform imaging each time the size of the object differs in the vertical direction. This makes it possible to perform imaging that is necessary and sufficient for three-dimensional reproduction.

[0040] [Modification 2] The first control unit 14 may move the imaging device 130 in a direction that includes at least a vertical component. For example, the first control unit 14 may control a multi-joint robot arm to which the imaging device 130 is provided to move the imaging device 130 in a direction that includes at least a vertical component.

[0041] [Other Modifications] The block diagrams used in the description of the above embodiments show functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may also be realized by combining software with the one or more of the above devices.

[0042] The processing procedures, sequences, flowcharts, etc., of each aspect / embodiment described in this disclosure may be reordered, provided they do not contradict each other. For example, the methods described in this disclosure present various step elements using exemplary order and are not limited to the specific order presented.

[0043] Functions include, but are not limited to, judgment, decision, determination, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, assumption, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (mapping), and assigning. For example, a functional block (configuration part) that enables transmission is called a transmitting unit or transmitter. In all cases, as mentioned above, the method of implementation is not particularly limited.

[0044] The notification of information is not limited to the aspects / embodiments described in the present disclosure, and other methods may be used. For example, the notification of information may be implemented by physical layer signaling (e.g., DCI (Downlink Control Information), UCI (Uplink Control Information)), upper layer signaling (e.g., RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, notification information (MIB (Master Information Block), SIB (System Information Block))), other signals, or combinations thereof. Also, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC connection setup (RRC Connection Setup) message, an RRC connection reconfiguration (RRC Connection Reconfiguration) message, etc.

[0045] Each aspect / embodiment described in the present disclosure may be applicable to at least one of systems using LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), and other suitable systems, as well as next-generation systems extended based on these. Also, multiple systems may be combined (e.g., a combination of at least one of LTE and LTE-A and 5G, etc.) and applied.

[0046] Information, etc. may be output from an upper layer (or a lower layer) to a lower layer (or an upper layer). It may be input and output via a plurality of network nodes.

[0047] The input and output information, etc. may be stored in a specific location (e.g., a memory), or may be managed using a management table. The input and output information, etc. may be overwritten, updated, or appended. The output information, etc. may be deleted. The input information, etc. may be transmitted to other devices.

[0048] The determination may be made based on a value represented by 1 bit (0 or 1), or based on a boolean value (true or false), or based on a numerical comparison (e.g., comparison with a predetermined value).

[0049] Each aspect / embodiment described in this disclosure may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of specific information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0050] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Therefore, the descriptions in the present disclosure are illustrative and not intended to be restrictive in any way.

[0051] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name. Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technologies (such as infrared or microwave), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0052] The information, signals, etc., described in this disclosure may be represented using any of the various different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof. Terms used in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). A signal may also be a message. Furthermore, a component carrier (CC) may also be called a carrier frequency, cell, frequency carrier, etc.

[0053] The terms “system” and “network” as used in this disclosure are interchangeable.

[0054] Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information. For example, radio resources may be indicated by an index. The names used for the parameters described above are not limiting in any way. Moreover, the formulas, etc., that use these parameters may differ from those expressly disclosed in this disclosure. Various channels (e.g., PUCCH, PDCCH, etc.) and information elements can be identified by any suitable name, and therefore, the various names assigned to these various channels and information elements are not limiting in any way.

[0055] In this disclosure, terms such as “Mobile Station (MS),” “user terminal,” “User Equipment (UE),” and “terminal” may be used interchangeably. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, pedestrian agent, mobile client, client, or several other appropriate terms.

[0056] The mobile device 10 may also be called a transmitting device, a receiving device, a communication device, etc.

[0057] The terms "determining" and "decision" can encompass a wide variety of actions. "Determining" and "decision" can include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in tables, databases, or other data structures), and ascertaining. Furthermore, "determining" and "decision" can also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory). Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

[0058] The terms “connected,” “coupled,” or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0059] In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0060] In the configuration of each of the above devices, "means" may be replaced with "part," "circuit," "device," etc.

[0061] Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

[0062] In this disclosure, if articles are added through translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0063] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different."

[0064] 10...Moving body, 1001...Processor, 1002...Memory, 1003...Storage, 1004...Communication device, 1005...User interface device, 1007...Up / down sliding mechanism, 1008...Wheel control mechanism, 11...Acquisition unit, 12...Storage unit, 13...Image recognition unit, 14...First control unit, 15...Second control unit, 100...Control device, 110...Main body, 120...Wheels, 130...Imaging device, 140...Support member, R...Road surface, O...Object, V...Vertical direction.

Claims

1. A control device comprising: an image recognition unit that performs image recognition processing on image data representing an image captured by an imaging device mounted on a moving body; a first control unit that moves the imaging device in a direction including at least a vertical component according to the result of the image recognition processing; and a second control unit that controls the movement of the moving body, the second control unit that stops the moving body at least when the imaging device is capturing an image.

2. The control device according to claim 1, characterized in that, when the first control unit recognizes image data corresponding to an object with a different shape or size in the vertical direction as a result of the image recognition process, it moves the imaging device in a direction that includes at least a vertical component.

3. The control device according to claim 2, characterized in that when the second control unit recognizes image data corresponding to an object with a different shape or size in the vertical direction as a result of the image recognition process, it moves the moving body in a direction that approaches the object.

4. The control device according to claim 1, characterized in that, when the first control unit recognizes image data corresponding to an object with a different shape or size in the vertical direction as a result of the image recognition process, it causes the imaging device to move by a different distance or number of times in a direction that includes at least a vertical component, according to the shape or size.

5. The control device according to claim 1, characterized in that the first control unit controls a mechanism that allows the imaging device to move along a member extending in the vertical direction, thereby moving the imaging device in the vertical direction.

6. The control device according to claim 1, characterized in that the first control unit controls the articulated robot arm on which the imaging device is provided to move the imaging device in a direction including at least a vertical component.

7. An information processing system comprising a control device according to any one of claims 1 to 6, and an information processing device that generates three-dimensional image data using two-dimensional image data showing images captured from multiple different viewpoints by the imaging device.

8. The information processing system according to claim 7, characterized in that the information processing device generates three-dimensional image data using Gaussian Splatting.

Images