Image recognition apparatus and image recognition method

Abstract

To efficiently assist appropriate component assembly work.SOLUTION: An image recognition apparatus 100 includes: a three-dimensional point cloud restoration unit 200 which restores a three-dimensional point cloud from an image captured by a camera, to calculate an image point-cloud positional relationship; an alignment unit 201 which executes alignment between the three-dimensional point cloud and three-dimensional design information; and a component assembly position specifying unit 202 which specifies a component assembly point-cloud position based on the three-dimensional design information, and specifies a component assembly position on the image, based on the image point-cloud positional relationship and the component assembly point-cloud position.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention generally relates to techniques for image recognition devices and image recognition methods, and more particularly to techniques that enable efficient support for accurate part assembly work. [Background technology]

[0002] In recent years, there has been a strong movement to utilize Augmented Reality (AR) technology to improve operational efficiency in the manufacturing industry. Many specific application examples of AR technology have been proposed, especially in large-scale parts assembly work sites where a huge number of parts are handled. One such example involves image recognition of each part and reading AR markers attached to the parts, thereby obtaining information on the part type, specifications, and assembly work on-site and utilizing this information for subsequent work.

[0003] As described above, a conventional technique for application to parts assembly work, etc., is proposed in Patent Document 1. Patent Document 1 discloses a technique relating to an information processing device, an information processing method, and a program that can appropriately map a photographed image of an object onto a CAD model.

[0004] The above technology relates to an information processing device including a control unit, which executes the following operations: acquiring a CAD model of an object; converting the CAD model into a CAD model point cloud; acquiring a plurality of images of the object; applying a 3D model creation algorithm to the plurality of images to acquire a point cloud and information indicating the position and orientation of a camera that captured the images; aligning the CAD model point cloud with the acquired point cloud to acquire a transformation matrix from the acquired point cloud to the CAD model point cloud; converting the information indicating the position and orientation of the camera into information indicating the position and orientation of the camera in CAD space using the transformation matrix; extracting degradation information of the object from the plurality of images; and using the information indicating the position and orientation of the camera in CAD space to assign the degradation information to the CAD model to acquire a degraded CAD model.

[0005] Furthermore, Patent Document 2 discloses a technique for improving the accuracy of measuring dimensions required for product inspection by using CAD data and measurement point cloud data at the time of product design.

[0006] This technology relates to a non-contact three-dimensional dimension measuring device that is communicatively connected to a drive unit that moves an object to be inspected translationally or rotationally, and to a laser scanner that scans the object by irradiating it with laser light, and that controls the drive unit and the laser scanner to measure dimensions of a portion of the object to be inspected, and that is characterized by having: a three-dimensional CAD data processing unit that adds dimension measurement data that identifies the portion whose dimensions are to be measured to as-designed three-dimensional CAD data of the object to be inspected; a point cloud data processing unit that generates point cloud data including three-dimensional coordinate values ​​of the points at which the laser light is irradiated on the object to be inspected based on the scanning results obtained by the laser scanner, and that overlays the three-dimensional CAD data with the dimension measurement data added on the point cloud data; and a dimension measurement processing unit that measures the dimensions of the portion identified by the dimension measurement data in the overlaid point cloud data. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] WO2022 / 259383 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-32380 Summary of the Invention [Problem to be solved by the invention]

[0008] However, depending on the part assembly environment, the above-mentioned conventional technologies may not function sufficiently or may be difficult to apply in the first place. For example, in situations where the locations where parts need to be assembled are widely dispersed, the equipment (e.g., cameras), personnel, or number of tasks required to perform image recognition or read AR markers for those parts is likely to increase. In such cases, overall work costs and efficiency may worsen, making it difficult to improve work efficiency. Furthermore, in locations where it is difficult to install such equipment or allocate personnel, the technology itself may not be applicable.

[0009] On the other hand, even if so-called markerless AR technology is adopted, it is necessary to take images / point clouds of parts and their assembly positions in advance, and if the number of parts is enormous, the effort and cost involved can become significant.

[0010] Therefore, the present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique that can efficiently support accurate part assembly work. [Means for solving the problem]

[0011] The present application includes multiple means for solving the above-mentioned problems, examples of which are as follows: To solve the above-mentioned problems, an image recognition device according to one aspect of the present invention comprises a 3D point cloud restoration unit that restores a 3D point cloud from an image captured by a camera and calculates an image point cloud positional relationship that indicates a positional relationship between the image and the 3D point cloud, a registration unit that executes a registration process between the 3D point cloud and 3D design information that indicates an assembly position of each part on an assembly target, and a part assembly position identification unit that identifies a part assembly point cloud position that is an assembly position on the 3D point cloud corresponding to the assembly position based on the 3D design information, and identifies a part assembly image position that is a part assembly position on the image based on the image point cloud positional relationship and the part assembly point cloud position.

[0012] In order to solve the above problem, an image recognition method according to one embodiment of the present invention is characterized in that an information processing device executes the following processes: reconstructing a three-dimensional point cloud from an image captured by a camera, and calculating an image point cloud positional relationship indicating the positional relationship between the image and the three-dimensional point cloud; performing a registration process between the three-dimensional point cloud and three-dimensional design information indicating the assembly position of each part on an assembly target; and specifying a part assembly point cloud position, which is an assembly position on the three-dimensional point cloud that corresponds to the assembly position, based on the three-dimensional design information, and specifying a part assembly image position, which is the part assembly position on the image, based on the image point cloud positional relationship and the part assembly point cloud position. [Effects of the Invention]

[0013] According to the present invention, accurate part assembly work can be efficiently supported. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of a network configuration including an image recognition device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image recognition apparatus according to the present embodiment. [Figure 3]FIG. 2 is a diagram illustrating an example of a functional configuration of an image recognition device according to the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of information held by the image recognition device according to the present embodiment. [Figure 5A] FIG. 1 is a diagram illustrating an example of the configuration of a parts management system according to an embodiment of the present invention. [Figure 5B] FIG. 2 is a diagram illustrating an example of the configuration of a part DB according to the present embodiment. [Figure 6A] FIG. 1 is a diagram illustrating an example of the configuration of a process control system according to an embodiment of the present invention. [Figure 6B] FIG. 2 is a diagram showing an example of the configuration of a process DB in the present embodiment. [Figure 7] 10 is a diagram showing an example of the configuration of assembly target three-dimensional structure information in this embodiment. FIG. [Figure 8] FIG. 4 is a diagram showing an example of the configuration of part assembly position information in the present embodiment. [Figure 9] FIG. 2 is a diagram showing an example of the configuration of a process parts correspondence table in the present embodiment. [Figure 10] FIG. 4 is a diagram showing an example of the configuration of a parts assembly list in the present embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of the configuration of part assembly point cloud position information according to the present embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of a flow of an image recognition method according to the present embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a warning output in the present embodiment. [Figure 14] FIG. 2 is a diagram illustrating an example of a flow of an image recognition method according to the present embodiment. [Figure 15] FIG. 2 is a diagram showing an example of a three-dimensional point cloud in this embodiment. [Figure 16] 10A and 10B are diagrams illustrating an example of a part installation image position in the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] In the following description, a communication device may be one or more communication interface devices, which may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., an NIC and an HBA (Host Bus Adapter)).

[0016] In the following description, a "main storage device" refers to one or more memory devices, which are an example of one or more storage devices. At least one of the memory devices in the main storage device may be a volatile memory device or a non-volatile memory device.

[0017] In the following description, an "auxiliary storage device" may be one or more persistent storage devices, which are an example of one or more storage devices. The persistent storage device may typically be a non-volatile storage device, specifically, for example, a hard disk drive (HDD), a solid state drive (SSD), or a non-volatile memory express (NVMe) drive.

[0018] Furthermore, in the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a hardware circuit that performs part or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0019] In the following description, information that provides an output in response to an input may be described using expressions such as "xxx table" or "xxx database." However, this information may be data of any structure (for example, structured data or unstructured data), or may be a learning model such as a neural network, genetic algorithm, or random forest that generates an output in response to an input. Therefore, "xxx table" or "xxx database" may be referred to as "xxx information." In the following description, the structure of each database or table is an example, and one database or table may be divided into two or more databases or tables, or all or part of two or more databases or tables may be one database or table.

[0020] In the following description, processing may be described using a "program" as the subject. However, because a program is executed by a processor to perform a predetermined process using a storage device and / or an interface device, etc., as appropriate, the subject of the process may also be the processor (or a device such as a controller having the processor). A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0021] In addition, in the following description, when describing elements of the same type without distinguishing between them, common parts of the reference symbols may be used, and when describing elements of the same type with distinction between them, reference symbols or element identifiers may be used.

[0022] <Network configuration including image recognition device> The present embodiment will be described below. FIG. 1 is a diagram showing an example of a network configuration including an image recognition device 100 of the present embodiment. In a network N, the minimum configuration that mainly implements the image recognition method of the present embodiment is the image recognition device 100. However, as in the network configuration shown in the figure, a configuration in which the image recognition method is executed in which the image recognition device 100 can cooperate with at least one of a management terminal 10, a parts management system 20, and a process management system 30 may also be adopted. In this case, the image recognition device 100 and a group of devices that cooperate with it may be defined as an image recognition system 1.

[0023] As an example, the image recognition device 100 in this embodiment may be a portable information processing device that appropriately links with a predetermined business system as needed to manage and support part assembly work at a manufacturing site. Such image recognition device 100 takes into account the technical background that it is difficult to appropriately maintain work costs and efficiency in situations where parts assembly locations are widely dispersed or where there are a huge number of parts. The image recognition device 100 in this embodiment is a device that can efficiently support accurate part assembly work.

[0024] The image recognition device 100 described above is specifically a so-called first-person camera device, and specific examples thereof include smart glasses, cameras with AR functions, tablets, and smartphones. Of course, such a camera device may be implemented in either a form that includes all of the functions and data required for the image recognition device 100 of this embodiment, or a form that is appropriately accessed and used from outside via a network N. In other words, if the network environment at the location where the image recognition device 100 is used is nonexistent or poor, the camera device may be implemented as a standalone image recognition device 100. If the network environment is good, the camera device may be implemented as a camera device that appropriately cooperates with an external system such as a parts management system 20 or a process management system 30 as shown in FIG. 1.

[0025] The network N connecting the image recognition device 100 with the management terminal 10, the component management system 20, and the process management system 30 may be, but is not limited to, the Internet, a local area network (LAN), a wide area network (WAN), or a mobile phone network. The image recognition device 100 is assumed to be an information processing device operated by a manufacturer that performs and manages component assembly work. Of course, the image recognition device 100 may also be operated by the worker who performs the component assembly work.

[0026] Among the devices connected to the network N, the management terminal 10 is a terminal for performing necessary settings and data storage for the image recognition device 100. The management terminal 10 is an information processing device operated by a manager or the like of the manufacturer company, and is specifically implemented as a personal computer (PC), tablet terminal, smartphone, or the like. The manager or the like operates the management terminal 10 to obtain, for example, 3D design information 300 and a process-component correspondence table 320 from the component management system 20 or the process management system 30, and sets these in the image recognition device 100 via the network N. Of course, such operations may be performed autonomously by the image recognition device 100. In this case, the management terminal 10 itself is not necessary.

[0027] Furthermore, the management terminal 10 may have a function to identify the position of the worker performing the part assembly work, i.e., the work position, at regular intervals, either autonomously or in response to a request from the image recognition device 100, at the site of the part assembly work, and notify the image recognition device 100 of this information. In this case, for example, the coordinates of the worker's location are identified based on the measurement results of various sensors (e.g., position sensors such as infrared sensors and ultrasonic sensors) using a coordinate system set at the work site as a reference. Of course, the management terminal 10 may also accept input from a manager or the like who is observing the position information of the worker during work, and respond to the image recognition device 100 with this as work position information.

[0028] Furthermore, the parts management system 20 is a system that manages parts used as targets for assembly work at the above-mentioned manufacturers and the like. For this reason, the parts management system 20 includes a parts DB 21 that stores management information for parts, as shown in Fig. 5A. As shown in Fig. 5B, this parts DB 21 is a collection of records that use a parts number that uniquely identifies a part as a key and contain values ​​such as the shape type, part name, and specifications (e.g., dimensions, weight, material, function, etc.) of the part. Of course, if the image recognition device 100 itself manages and holds information similar to the information stored in the parts DB 21, the parts management system 20 would not be necessary.

[0029] The process control system 30 is a system for managing the processes of part assembly work carried out by the above-mentioned manufacturers, etc. Therefore, the process control system 30 is provided with a process DB 31 that stores process management information, as shown in Fig. 6A. As shown in Fig. 6B, this process DB 31 is a collection of records that use a process number that uniquely identifies a process as a key, and that contain values ​​such as the content of the process, operation number, operation content, work location, parts used, and tools used.

[0030] Such information in the process DB 31 is notified to the image recognition device 100 from the process control system 30, and, for example, the status (assembly situation) of each task in each process, i.e., part assembly, is reflected in a part assembly list (described later in FIG. 10). Of course, if the image recognition device 100 itself manages and holds information similar to the information stored in such process DB 31, the process control system 30 becomes unnecessary.

[0031] The process control system 30 has a function of detecting the start and end of a process based on a notification (e.g., from a manager or worker) from the management terminal 10 or the image recognition device 100, and managing the progress of the process. Therefore, the process control system 30 can manage the status of each process in the process DB 31 based on the notification. The process control system 30 may notify the image recognition device 100 or the management terminal 10 of the status of each process as required / necessary.

[0032] On the other hand, the image recognition device 100 executes the following processes: reconstructing a three-dimensional point cloud from an image captured by an imaging device that is a first-person camera, calculating an image point cloud positional relationship that indicates the positional relationship between the image and the three-dimensional point cloud, aligning the three-dimensional point cloud with three-dimensional design information that indicates the assembly position of each part on an assembly target, specifying a part assembly point cloud position that is an assembly position on the three-dimensional point cloud that corresponds to the assembly position based on the three-dimensional design information, and specifying a part assembly image position that is a part assembly position on the image based on the image point cloud positional relationship and the part assembly point cloud position. Note that the specific configuration and functions of the image recognition device 100 will be described in detail later.

[0033] Data exchange between the image recognition device 100 and each of the management terminal 10, parts management system 20, and process management system 30 may be performed according to, for example, an API (Application Programming Interface) protocol. In this case, it is assumed that each device is pre-implemented with the functions and configurations for executing each process of requests and responses by the API.

[0034] <Hardware configuration of image recognition device> 2 is a diagram showing an example of the hardware configuration of the image recognition device 100 in this embodiment. However, the image recognition device 100 can also be considered as an information processing device corresponding to each device other than the image recognition device 100 that configures the image recognition system 1. The image recognition device 100 is configured such that a processor 101, a main memory device 102, an auxiliary memory device 103, an input device 104, an output device 105, a communication device 106, an image capture device 107, and a position measurement device 108 are communicatively connected by an appropriate interface such as an internal BUS.

[0035] Of these, the auxiliary storage device 103 is a storage means configured with non-volatile storage elements as already described, and in this embodiment, it stores at least information such as 3D design information 300 (configured from assembly object 3D structure information 310 and part assembly position information 311) and a process-part correspondence table 320, which will be described later with reference to Figures 4 and 7 to 9. Specific data configuration examples of this information will be described later.

[0036] The main memory device 102 also serves as a storage means for storing a program 1021, including an OS (Operating System) that controls the image recognition device 100 as a whole, various applications, and the like. The processor 101 loads and executes the program 1021 in the main memory device 102, thereby implementing required functions. The main memory device 102 also stores at least each piece of information, such as a component assembly list 400 and component assembly point cloud position information 410 (similar in configuration to the component assembly position information 311), shown in Fig. 4 and Fig. 10. Specific examples of the data configuration of this information will be described later.

[0037] The input device 104 is a means for accepting input operations by an operator of the image recognition device 100, and can be a UI (User Interface) such as a touch panel, a microphone, or an eye tracking function. The output device 105 is a means for outputting the results of information processing to the operator of the image recognition device 100, and can be a UI (User Interface) such as a display or a speaker.

[0038] The communication device 106 is a communication means configured by a communication chip compatible with the protocol of the network N. The communication device 106 accesses the network N and is communicatively connected via this network N to other devices that can cooperate with the image recognition device 100 (such as the management terminal 10, the parts management system 20, and the process management system 30).

[0039] As already mentioned, the image capturing device 107 is a first-person camera unit, and is controlled by a function implemented by the execution of the program 1021, and has an image capturing function that is the core of the image recognition device 100, i.e., the smart glasses. Specifically, the image capturing device 107 is configured with a general digital camera, or a LiDAR (Light Detection and Ranging), a stereo camera, a ToF (Time of Flight) camera, or the like. When the image capturing device 107 is a general digital camera, the image recognition device 100 acquires a group of images of a subject (in this case, a part assembly target or a part) by continuous shooting with the image capturing device 107, and generates a 3D point cloud by processing the images with a 3D point cloud (3D structure) restoration library or the like.

[0040] Such an imaging device 107 can be said to be a device that captures images of a part and an object to which the part is to be assembled at the site of part assembly work, acquires images, and provides the images for restoring a 3D point cloud, alignment, etc., which will be described later. In practice, by associating the coordinate values ​​(values ​​in the coordinate system in the captured image) of each location indicated by the 3D point cloud obtained as described above with absolute position information (for example, GPS coordinate values, etc.) of the part and its object to be assembled, obtained by a position measurement device 108 (described in detail later), the resulting data becomes 3D point cloud data that can be utilized by the image recognition device 100 of this embodiment.

[0041] Furthermore, the position measurement device 108 is a unit that measures the position information of the image recognition device 100, or the worker performing the part assembly work, or the part or the object to which the part is to be assembled, and is configured with general units for position measurement, such as a GPS unit, a beacon unit, an infrared sensor, an ultrasonic sensor, etc. However, the implementation form of the position measurement device 108 is not limited to the above example, and it is also possible to adopt an implementation form that recognizes position markers (or specific objects, specific structures, etc. registered in advance to function as position markers) installed at various locations on the site or at predetermined positions on the object to which the part is to be assembled from images of the surrounding environment captured by the imaging device 107, and determines the position information.

[0042] <Functional configuration of image recognition device> Next, the functional configuration of the image recognition device 100 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the functional configuration of the image recognition device 100 in this embodiment. The image recognition device 100 has functional units, including a 3D point cloud restoration unit 200, a registration unit 201, a part assembly position identification unit 202, a point cloud resampling unit 203, a part determination unit 204, a warning output unit 205, and a process determination unit 206. Each of these functional units is implemented by the processor 101 executing a program 1021. In addition, in order for each of these functional units to function effectively and perform the necessary processing, the image recognition device 100 also has various information such as 3D design information 300 and a process-part correspondence table 320.

[0043] Of these, the 3D point cloud restoration unit 200 is a functional unit that restores a 3D point cloud from an image captured by the image capture device 107 and calculates an image point cloud positional relationship that indicates the positional relationship between the image and the 3D point cloud. To this end, the 3D point cloud restoration unit 200 controls the image capture device 107 and position measurement device 108 already described to generate a 3D point cloud of a subject, such as a part assembly target or a part. Specifically, the 3D point cloud restoration unit 200 includes a library for point cloud restoration and an engine that executes restoration processing by applying the image group obtained from the image capture device 107 to the library. During restoration processing, the 3D point cloud restoration unit 200 processes each pixel in the image using a predetermined algorithm to obtain a point cloud, and therefore can naturally recognize the correspondence between each location in the image and the 3D point cloud restored from it.

[0044] In the restoration process, the position information of the target location obtained from the position measurement device 108 is used to identify and link the positions of each point in the 3D point cloud, and generate the final 3D point cloud. The configuration and functions of the 3D point cloud restoration unit 200 can be developed and implemented from scratch, or can be implemented by appropriately adopting existing tools and applications and installing or calling them into the image recognition device 100.

[0045] The registration unit 201 is a functional unit that executes registration processing between the above-mentioned 3D point cloud and 3D design information 300. The 3D design information 300 is 3D CAD data that indicates the installation position of each component on the installation target, and is composed of installation target 3D structure information 310 and component installation position information 311, as shown in FIG. 4. Details of each piece of information will be described later. The registration processing is a process of comparing the position information of each point in the 3D point cloud with the position information of each component and each installation target indicated by the 3D design information 300, identifying a plausible point cloud for the component or its installation target (such as a component), and determining the correspondence between them. The registration function itself can be achieved by appropriately adopting existing technology.

[0046] Note that when aligning the 3D point clouds, the following known registration methods are executed as needed. A 3D point cloud based on images captured by the image capture device 107 is generated only within the range of the angle of view (field of view) of the images. Therefore, depending on the direction, position, and number of images captured, many blind spots may occur in the generated point cloud. Therefore, in order to generate a point cloud by processing a group of images captured by multiple images and to generate a point cloud for the entire subject without omissions, it is necessary to register the point clouds generated from each of the multiple images and expand them into a point cloud that constitutes the entire subject.

[0047] Registration techniques involve finding correspondences between two or more 3D point clouds and estimating a transformation to convert those point clouds into a common coordinate system. More specifically, this technique is called iterative closest point (ICP). Given a reference point cloud and a target point cloud, ICP aligns the reference point cloud and the target point cloud by minimizing the distance between them, i.e., estimating the optimal rotation and translation. This method calculates corresponding points between the reference point cloud and the target point cloud, calculates a transformation that minimizes the distance between those corresponding points, and applies this transformation to the target point cloud. Of course, the above technique is merely an example. Other possible techniques include a probability-based method formulated as a maximum likelihood (ML) estimation problem using the GMM method, or a registration method using a state space model (SSM) (Surface-Based Registration with a Particle Filter).

[0048] The part assembly position specifying unit 202 is a functional unit that specifies part assembly point cloud position information 410 (see FIG. 11 ), which is an assembly position on a 3D point cloud corresponding to the part assembly position (a value indicated by the part assembly position information 311), based on the 3D design information 300, and specifies a part assembly image position, which is a part assembly position on an image, based on the image point cloud position relationship (specified by the 3D point cloud restoration unit 200) and the part assembly point cloud position information 410. In this case, the part assembly position specifying unit 202 can specify the part assembly position on an image, i.e., the part assembly image position, by comparing the point cloud information 410 (see FIG. 11 ), which corresponds to the part assembly position among the 3D point clouds and which has been determined by processing in the registration unit 201, with the positions of each point on the image from which the 3D point cloud was generated.

[0049] The point cloud resampling unit 203 is a functional unit that generates a mesh from a 3D point cloud and generates a resampled and restored 3D point cloud by resampling the point cloud from the mesh. The mesh can be generated using general dedicated software. The concept of mesh generation can be, but is not limited to, generating triangular areas connecting points in the 3D point cloud, setting surfaces in the triangular areas, and generating polygon data as a collection of these surfaces.

[0050] In the process of generating such meshes, the dedicated software performs processes such as so-called hole filling (holes occur where point clouds cannot be generated for areas that are not on the surface of the subject in the image, but are located deep within the subject, or in blind spots in the image, and these holes are filled in), bridging (the process of forming a shape by connecting the edges of each facet of the triangular area), and repair, and accurately converts the 3D shapes of parts and the objects to which they are to be assembled into data.

[0051] Furthermore, point cloud resampling unit 203 generates a 3D design information point cloud by sampling a point cloud from 3D design information 300. This 3D design information point cloud is generated as a point cloud uniformly distributed on each face and each vertex of a 3D shape that can be modeled based on information such as the structure, size, orientation, and position indicated by assembly target 3D structure information 310 and part assembly position information 311. Meanwhile, in this case, alignment unit 201 aligns the 3D point cloud with 3D design information 300 using the resampled and restored 3D point cloud and the 3D design information point cloud.

[0052] The part determination unit 204 is a functional unit that performs image recognition on a portion of the image that includes at least the part assembly position on the image identified by the part assembly position identification unit 202, and determines at least one of the presence or absence and correctness of the part at that position. For this purpose, the part determination unit 204 stores information regarding the image feature amounts of each part in advance, or references and uses the information from the part DB 21 of the part management system 20 as needed. The part determination unit 204 sets the result of the determination, for example, in the "assembly status" column of the part assembly list 400 (see FIG. 10). In other words, the part assembly status is accumulated as information that can be utilized.

[0053] The warning output unit 205 is a functional unit that outputs a predetermined warning to the output device 105, or the management terminal 10 or process control system 30 on the network N, depending on the result of the judgment by the part judgment unit 204. The result of the judgment is a value such as the presence or absence of the target part, or whether it is correct or incorrect, and the warning is a message that explains the event, such as the presence or absence of the target part, or whether it is correct or incorrect.

[0054] It is preferable that the warning output unit 205 outputs a warning when a part is missing, among the above warnings, in response to detection of the end of a predetermined work process. Therefore, the warning output unit 205 outputs the above warning when, for example, it receives a notification of the end of a process from a worker or a manager via the input device 104 or the management terminal 10, or when the process determination unit 206 determines that the process has ended.

[0055] The warning output unit 205 also has a function of, in response to detecting an event in which the assembly work position has deviated from the specified position, referring to the above-mentioned determination result (the value in the assembly status column of the part assembly list 400), and outputting a warning to that effect if it is determined that a part is not present at the part assembly position. In this case, the warning output unit 205 obtains its own position information from the position measurement device 108 regarding the above-mentioned separation event, and also obtains the process status from the process determination unit 206, the management terminal 10, the input device 104, or the process DB 31 of the process management system 30. If the status is "completed" (or "in progress") and the position information indicates that the part is away from the specified position on the site by a certain distance, the warning output unit 205 executes processing related to the above-mentioned warning.

[0056] The process determination unit 206 is a functional unit that manages and updates the status of each process based on information in the process DB 31 of the process management system 30 or information received from a manager or the like via the management terminal 10 or input device 104. Therefore, the process determination unit 206 holds the status of each process in the main memory device 102 for a certain period of time or until the process is completed, for example.

[0057] <Example of information configuration> Next, a specific example of the configuration of information managed by the image recognition device 100 of this embodiment and used in various processes will be described. Fig. 7 is a diagram showing an example of the configuration of assembly target three-dimensional structure information 310 in this embodiment. This assembly target three-dimensional structure information 310 is a collection of records in which values ​​such as the attribute, name, size, orientation, position of the assembly target three-dimensional structure and the number of the part to be assembled are linked together using an ID that uniquely identifies the assembly target three-dimensional structure as a key.

[0058] 8 is a diagram showing an example of the configuration of the part assembly position information 311 in this embodiment. The part assembly position information 311 is a collection of records in which values ​​such as the number of definitions, shape type, size, orientation, and position of the part are linked together using a part number that uniquely identifies the part as a key.

[0059] 9 is a diagram showing an example of the configuration of the process and parts correspondence table 320 in this embodiment. The process and parts correspondence table 320 is a collection of records in which, using a process number that uniquely identifies a process as a key, values ​​such as a task number indicating each task included in the process, a component number of a component used in the task, and a component name are linked together.

[0060] 10 is a diagram showing an example of the configuration of a part assembly list 400 in this embodiment. The part assembly list 400 is a collection of records that use a process number that uniquely identifies a process as a key to link together values ​​such as the operation number of the work included in the process, the part number of the part to be assembled in the work, the part name, and the assembly status.

[0061] 11 is a diagram showing an example of the configuration of the part assembly point cloud position information 410 in this embodiment. The part assembly point cloud position information 410 is a collection of records that use an ID that uniquely identifies the 3D structure to be assembled as a key, and link the values ​​of the assembly position of the part to the structure, the target point cloud, the position (of the structure itself), and the part to be assembled.

[0062] <Image Recognition Method: Main Flow> Next, a processing flow in the image recognition method of this embodiment will be described. Fig. 12 is a diagram showing an example of a flow in the image recognition method of this embodiment. Here, it is assumed that a worker wearing the image recognition device 100, which is, for example, smart glasses, performs part assembly work in a factory of a certain manufacturer. This part assembly work involves assembling parts that constitute the product or parts necessary for product manufacturing, with the product being the target of assembly being a part or all of its structure.

[0063] First, the worker uses the photographing device 107 of the image recognition device 100 worn or held by the worker to photograph an object to which parts are to be installed and the parts to be installed in the object as subjects. The image recognition device 100 acquires multiple images by photographing (S1).

[0064] Next, the 3D point cloud restoration unit 200 of the image recognition device 100 restores a 3D point cloud (see 3D point cloud G10 in FIG. 15) for the multiple images obtained in S1 above by, for example, applying it to a library and engine for point cloud restoration (S2). Through this restoration operation, the 3D point cloud restoration unit 200 calculates an image point cloud positional relationship that indicates the positional relationship between the multiple images obtained by shooting in S1 and the 3D point cloud.

[0065] In the above restoration process, the 3D point cloud restoration unit 200 uses position information about the target location obtained from the position measurement device 108 to identify and link the positions of each point in the 3D point cloud and generate a final 3D point cloud. The configuration and functions of the 3D point cloud restoration unit 200 can be developed and introduced from scratch, or can be implemented by appropriately adopting existing tools and applications and installing or calling them into the image recognition device 100.

[0066] Next, the alignment unit 201 of the image recognition device 100 executes alignment processing between the above-mentioned three-dimensional point cloud and three-dimensional design information 300 (see FIG. 4) (S3). This alignment processing is processing to compare position information of each point in the three-dimensional point cloud with position information of each component or each assembly target indicated by the three-dimensional design information 300, identify point clouds in the three-dimensional point cloud that are likely to be components or their assembly targets (such as members), and determine the correspondence between the three-dimensional design information 300 and the three-dimensional point cloud.

[0067] Note that when aligning the 3D point clouds, the following known registration methods are executed as needed. A 3D point cloud based on images captured by the image capture device 107 is generated only within the range of the angle of view (field of view) of the images. Therefore, depending on the direction, position, and number of images captured, many blind spots may occur in the generated point cloud. Therefore, in order to generate a point cloud by processing a group of images captured by multiple images and to generate a point cloud for the entire subject without omissions, it is necessary to register the point clouds generated from each of the multiple images and expand them into a point cloud that constitutes the entire subject.

[0068] Registration techniques involve finding correspondences between two or more 3D point clouds and estimating a transformation to convert those point clouds into a common coordinate system. More specifically, this technique is called iterative closest point (ICP). Given a reference point cloud and a target point cloud, ICP aligns the reference point cloud and the target point cloud by minimizing the distance between them, i.e., estimating the optimal rotation and translation. This method calculates corresponding points between the reference point cloud and the target point cloud, calculates a transformation that minimizes the distance between those corresponding points, and applies this transformation to the target point cloud. Of course, the above technique is merely an example. Other possible techniques include a probability-based method formulated as a maximum likelihood (ML) estimation problem using the GMM method, or a registration method using a state space model (SSM) (Surface-Based Registration with a Particle Filter).

[0069] Next, the image recognition device 100 executes the following processes S5 to S13 for each part. Then, the part assembly position specifying unit 202 of the image recognition device 100 specifies, based on the above-mentioned 3D design information 300, part assembly point cloud position information 410 (see part assembly point cloud position G15 in FIGS. 11 and 15) indicating the assembly position on the 3D point cloud that corresponds to the assembly position of the part (the value indicated by the part assembly position information 311) (S5).

[0070] Preferably, the part assembly position specifying unit 202 specifies the parts to be used in the work process specified by the management terminal 10 or the input device 104 based on the process-part correspondence table 320 (see FIG. 9), and specifies the part assembly position on the captured image for the parts. As for which process the above processing should be performed for, the process determination unit 206 specifies the status of the process based on information in the process DB 31 of the process management system 30 or information received from a manager or the like via the management terminal 10 or the input device 104, and sets the process as the target for handling.

[0071] In addition, the part assembly position identification unit 202 identifies the part assembly image position (see part assembly image position G25 in Figure 16), which is the part assembly position on the captured image G20 obtained in S1, based on the image point cloud position relationship (identified by the 3D point cloud restoration unit 200) and the part assembly point cloud position information 410 obtained in S5 (S6).

[0072] The part assembly position identification unit 202 can identify the part assembly position on the image, i.e., the part assembly image position, by comparing the point group information 410 (see FIG. 11) corresponding to the part assembly position in the 3D point group, which has been determined by the processing (S3) in the alignment unit 201, with the position of each point in the image from which the 3D point group was derived.

[0073] The above-mentioned 3D point cloud restoration process may be performed by the point cloud resampling unit 203. In this case, the point cloud resampling unit 203 generates a mesh from the 3D point cloud, and then generates a resampled restored 3D point cloud by resampling the mesh. The point cloud resampling unit 203 may generate this mesh by calling general dedicated software.

[0074] Furthermore, the point cloud resampling unit 203 generates a 3D design information point cloud by sampling a point cloud from the 3D design information 300. This 3D design information point cloud is generated as a point cloud uniformly distributed on each face and vertex of a 3D shape that can be modeled based on information such as the structure, size, orientation, and position indicated by the assembly target 3D structure information 310 and the part assembly position information 311. Meanwhile, the registration unit 201 in this case uses the resampled restored 3D point cloud and the 3D design information point cloud to perform registration between the 3D point cloud and the 3D design information 300. By performing this correspondence, a point cloud uniformly distributed on the object (such as a part or its assembly target structure) is obtained and compared with the 3D design information 300, enabling efficient and accurate registration.

[0075] Next, the part determination unit 204 of the image recognition device 100 performs image recognition on a portion of the image that includes at least the part assembly position on the image identified by the part assembly position identification unit 202, and determines whether or not a part is present at that position. To this end, the part determination unit 204 either stores information regarding the image features of each part in advance, or references and uses the information from the part DB 21 of the part management system 20 as needed. The part determination unit 204 sets the result of the determination, for example, in the "assembly status" column of the part assembly list 400 (see FIG. 10). In other words, the part assembly status is accumulated as information that can be utilized.

[0076] If the result of the above determination is that a part is not installed at the part installation position on the captured image (S8: N), image recognition device 100 ends this flow. On the other hand, if the result of the above determination is that a part is installed at the part installation position on the captured image (S8: Y), part determination unit 204 determines whether the part is an appropriate part that should have been installed (S9). In this determination, part determination unit 204 compares information on the feature amounts of the image of each part (stored in advance or referenced from part DB 21 of part management system 20) with the feature amounts of the part (part visible as an image) obtained by image recognition of the captured image, and determines whether the part is appropriate.

[0077] If the result of the above determination is that an inappropriate part has been assembled (S10: N), the part determination unit 204 stores the determination result, "incorrect," in the "assembly status" column of the part assembly list 400 (S11). Furthermore, the warning output unit 205 outputs the result of the determination by the part determination unit 204, i.e., a warning notice (see screen M10 in FIG. 13) notifying of the assembly of an incorrect part, to the output device 105, or to the management terminal 10 or process control system 30 on the network N (S12).

[0078] On the other hand, if the result of the above judgment S9 is that the appropriate parts have been assembled (S10: Y), the part judgment unit 204 stores the judgment result "correct" in the "assembly status" column of the part assembly list 400 (S13).

[0079] Furthermore, the warning output unit 205 performs the process shown in Fig. 14 in response to the detection of an event in which the position where the assembly work is being performed has moved away from the specified position. The above determination result (the value in the assembly status column of the part assembly list 400) is referenced to determine whether or not there are any parts for which assembly has not been completed (S20). If there are no parts, the process ends as is, but if there are parts, a warning is output indicating that there are no parts, i.e., that a part is missing (S21). An example of a warning screen for a missing part is shown in Fig. 13. For example, the screen displays information such as the process number, operation number, assembly target structure, and part, along with the fact that a part has been missing.

[0080] For the above-mentioned separation event, the warning output unit 205 obtains its own position information from the position measurement device 108, and also obtains the process status from the process determination unit 206, the management terminal 10, the input device 104, or the process DB 31 of the process management system 30. If the status is "completed" (or "in progress") and the position information indicates that the device is a certain distance away from the specified position on site, the warning output unit 205 executes the processing shown in FIG. 14 above.

[0081] It is preferable that the warning output unit 205 refers to the process-part correspondence table 320, and when it is determined that a part that should have already been performed at a certain point in time among a series of part assemblies that should be performed sequentially in a specific work process exists but is missing from its part assembly position based on the determination results (value in the assembly status column) accumulated in the part assembly list 400, it outputs a warning to that effect.

[0082] Although one embodiment of the present invention has been described above, this is merely an example for explaining the present invention, and the scope of the present invention is not limited to this embodiment. The present invention can be implemented in various other forms.

[0083] The above description can be summarized as follows: The following summary may include supplementary explanations and explanations of variations of the above description.

[0084] In the image recognition device 100 of this embodiment, a process-parts correspondence table 320 indicating the parts used in each work process may be held in a storage device, and the part assembly position identification unit may identify the parts used in a specified work process based on the process-parts correspondence table, and identify the part assembly positions on the image for the identified parts.

[0085] This eliminates the need for the image recognition device 100 to constantly store various pieces of information for identifying part assembly positions for all parts and their assembly targets, and allows for an operation mode in which only the data set related to the relevant work process is stored and used, thereby enabling more efficient support for accurate part assembly work.

[0086] Furthermore, the image recognition device 100 of this embodiment may further include a part determination unit and a warning output unit, wherein the part determination unit performs image recognition on a portion of the image that includes at least the identified part assembly position on the image, and determines at least one of the presence or absence and correctness of the part at that position, and the warning output unit outputs a predetermined warning according to the result of the determination.

[0087] This makes it possible to effectively improve the accuracy of the part assembly work based on the correctness or precision of the part assembly work, thereby more efficiently supporting accurate part assembly work.

[0088] Furthermore, in the image recognition device 100 of this embodiment, the part determination unit may store the determination result, which is the result of the determination, in a storage device, and the warning output unit may refer to the determination result stored in the storage device in response to detecting the end event of a predetermined work process, and if it is found that the part is missing or an incorrect part is present at the part assembly position, output a warning to that effect.

[0089] This effectively avoids situations in which workers are forced into a complicated and inefficient situation, such as when warnings are issued indiscriminately during or before the start of a work process, even when part assembly work has not yet begun. Ultimately, it becomes possible to more efficiently support accurate part assembly work.

[0090] Furthermore, in the image recognition device 100 of this embodiment, the warning output unit may refer to the judgment results accumulated in the storage device in response to detection of an event in which the position where the assembly work is performed has moved away from a specified position, and if it is found that the part is missing or an incorrect part is present at the part assembly position, output a warning to that effect.

[0091] This allows the system to automatically recognize the timing when the worker finishes the assembly work and leaves the site as the completion of the work, without any special instructions or notices from the worker, and to execute the various processes related to the warning. This ultimately makes it possible to more efficiently support accurate part assembly work.

[0092] Furthermore, in the image recognition device 100 of this embodiment, the warning output unit may refer to the judgment results accumulated in the storage device regarding a part assembly that should have already been performed at a certain point in time among a series of part assembly steps that should be performed sequentially in a specified work process, and if it is found that the part is missing or an incorrect part is present at the part assembly position, output a warning to that effect.

[0093] This makes it possible to accurately identify a process in a series of work steps where a task has been omitted due to a worker's carelessness or other reasons, and to prompt the worker to take action, thereby enabling more efficient support for accurate part assembly work.

[0094] Furthermore, the image recognition device 100 of this embodiment may further include a point cloud resampling unit that generates a mesh from the 3D point cloud and generates a resampled and restored 3D point cloud from the mesh by resampling the point cloud, wherein the point cloud resampling unit generates a 3D design information point cloud from the 3D design information by sampling the point cloud, and the alignment unit aligns the 3D point cloud with the 3D design information using the resampled and restored 3D point cloud and the 3D design information point cloud.

[0095] This makes it possible to efficiently generate a highly accurate 3D point cloud and use it for subsequent processing, thereby more efficiently supporting accurate part assembly work. [Explanation of symbols]

[0096] N Network 1. Image Recognition System 10 Management terminal 11 processors 12 Main storage 13 Auxiliary storage device 14 Input Devices 15 Output Devices 16. Communications equipment 17 Imaging equipment 18 Position measurement device 20 Parts Management System 21 Parts DB 30 Process Control System 31 Process DB 100 Image recognition device 101 processors 102 Main storage 103 Auxiliary storage device 110 Image capturing device 120 Display device 130 Communication equipment 200 3D point cloud reconstruction unit 201 Alignment section 202 Parts assembly position identification unit 203 Point Cloud Resampler 210 Parts Assembly Specific Department 211 Parts Judgment Unit 220 Warning Output Unit 221 Process Judgment Department 200 3D point cloud reconstruction unit 201 Alignment section 202 Parts assembly position identification unit 203 Point Cloud Resampler 204 Parts Assembly Specific Department 205 Parts Judgment Unit 206 Warning Output Unit 207 Process Judgment Department 300 3D design information 310 3D structural information of assembly target 311 Parts assembly position information 320 Process Parts Correspondence Table 330 Parts Assembly List G10 3D point cloud G15 Part assembly point cloud position G20 Photo G25 Parts assembly image position

Claims

1. a three-dimensional point cloud restoration unit that restores a three-dimensional point cloud from an image captured by a camera and calculates an image point cloud positional relationship that indicates a positional relationship between the image and the three-dimensional point cloud; a registration unit that executes registration processing between the three-dimensional point cloud and three-dimensional design information that indicates the assembly position of each part in an assembly target; a part assembly position specifying unit that specifies a part assembly point cloud position on the three-dimensional point cloud that corresponds to the assembly position based on the three-dimensional design information, and that specifies a part assembly image position that is the part assembly position on the image based on the image point cloud positional relationship and the part assembly point cloud position; An image recognition device comprising:

2. A process-parts correspondence table indicating the parts used in each work process is stored in a storage device; the part assembly position specifying unit specifies parts to be used in the specified work process based on the process-part correspondence table, and specifies part assembly positions on the image for the specified parts.

2. The image recognition device according to claim 1 .

3. Further comprising a part determination unit and a warning output unit, the part determination unit performs image recognition on a portion of the image including at least the specified part assembly position on the image, and determines at least one of the presence or absence and correctness of the part at the position; the warning output unit outputs a predetermined warning in accordance with the result of the determination.

2. The image recognition device according to claim 1 .

4. the part determination unit stores a determination result in a storage device; the warning output unit, in response to detection of an end event of a predetermined work process, refers to the judgment results accumulated in the storage device, and if it is found that the part is missing or an incorrect part is present at the part assembly position, outputs the warning to that effect.

4. The image recognition device according to claim 3.

5. The warning output unit In response to detection of an event in which the position where the assembly work is performed has moved away from a specified position, the determination results stored in the storage device are referenced, and if it is found that the part is missing or an incorrect part is present at the part assembly position, a warning to that effect is output.

4. The image recognition device according to claim 3.

6. The warning output unit With regard to a part assembly that should have already been performed at a certain point in time among a series of part assembly steps that should be performed sequentially in a predetermined work process, the determination results stored in the storage device are referenced, and if it is found that the part is missing or an incorrect part is present at the part assembly position, a warning to that effect is output.

4. The image recognition device according to claim 3.

7. a point cloud resampling unit that generates a mesh from the three-dimensional point cloud and generates a resampled three-dimensional point cloud from the mesh by resampling the point cloud; the point cloud resampling unit generates a three-dimensional design information point cloud by sampling the point cloud from the three-dimensional design information; the alignment unit aligns the three-dimensional point cloud and the three-dimensional design information using the resampled three-dimensional point cloud and the three-dimensional design information point cloud.

2. The image recognition device according to claim 1 .

8. The information processing device A process of restoring a three-dimensional point cloud from an image captured by a camera and calculating an image point cloud positional relationship that indicates a positional relationship between the image and the three-dimensional point cloud; A process of performing a registration process between the three-dimensional point cloud and three-dimensional design information indicating the assembly position of each part in an assembly target; a process of specifying a part assembly point cloud position, which is an assembly position on the three-dimensional point cloud corresponding to the assembly position, based on the three-dimensional design information, and specifying a part assembly image position, which is a part assembly position on the image, based on the image point cloud positional relationship and the part assembly point cloud position; An image recognition method comprising:

Images