Position and orientation determination of modules during construction of a modular industrial plant

Abstract

A machine vision system is provided for determining positions and orientations of modules during construction of a modular industrial plant in a defined volume to be occupied by the plant. The system includes an array of cameras having overlapping fields of view such that, when the defined volume is empty of modules, each position within the defined the volume is observable by plural of the cameras. The system further includes a computer system which, as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, is programmed to repeatedly: receive images from the cameras, and determine therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras.

Description

[0001] 008812141

[0002] 1

[0003] POSITION AND ORIENTATION DETERMINATION OF MODULES DURING CONSTRUCTION OF A MODULAR INDUSTRIAL PLANT

[0004] This application claims priority from EP 24198294.1 3 filed September 2024, the contents and elements of which are herein incorporated by reference for all purposes.

[0005] Field of the Invention

[0006] The present invention relates to determination of positions and orientations of modules during construction of a modular industrial plant.

[0007] Background

[0008] Conventionally, high capital costs are associated with the construction and operation of large industrial plants. Due to these high capital costs, there is interest in moving towards the development of plants constructed from smaller modular units.

[0009] In the modular approach, much of the plant can be factory-fabricated and delivered by road, rail or boat as modules to the prepared site. Once on site, the transported modules are then connected together to form the complete plant. This reduces costs, lead-times and risk, and provides quality improvements associated with the factory fabrication of the individual modules.

[0010] Each module may consist of an open (e.g. rectangular cuboid) frame into which equipment items such as apparatuses, pipework etc. are mounted. When the frame is moved on site into final position, attachments and connections are made to these items from other modules through the open sides of the frame.

[0011] For example, a proposal for a small modular reactor (SMR) advanced by Rolls-Royce SMR Limited has around 1500 standard transportable modules that will be manufactured and tested in off-site factories, and then assembled on location within a compact site footprint.

[0012] During plant construction, it is necessary to ensure that the modules are correctly and accurately located in their final positions so that adjacent modules can be correctly operatively connected.

[0013] Multi-camera systems which use stereovision principles to conduct point measurements are known. For example, US 2018 / 0343421 proposes a system that applies stereovision principles, alongside other techniques such as CAD models combined with single camera views, to measure the position of static objects. However, for motion tracking applications crossing multiple different camera fields of view (FOVs), US 2018 / 0343421 proposes the use of other measurement methods, such as single camera views combined with CAD files and markings of known dimensions.

[0014] The present invention has been devised in light of these considerations. 008812141

[0015] 2

[0016] Summary of the Invention

[0017] It would be desirable to provide a stereovision or multivision system (multivision being an extension of stereovision in which plural, parallel stereovision calculations are performed using images from three or more cameras) that can be used to guide construction of a modular industrial plant. Such a system could advantageously enable precise and high-speed tracking of points over wide distances (i.e. on the scale of an industrial plant) across multiple camera FOVs.

[0018] In general terms, the present invention relates to a machine vision system for assisting the correct installation and connection of modules of a modular industrial plant.

[0019] In a first aspect, the present invention provides a machine vision system for determining positions and orientations of modules during construction of a modular industrial plant in a defined volume to be occupied by the plant, the system including: an array of cameras having overlapping fields of view such that, when the defined volume is empty of modules, each position within the defined the volume is observable by plural of the cameras, and a computer system which, as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, is programmed to repeatedly: receive images from the cameras, and determine therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras.

[0020] As the positions of the cameras can be predefined, and as each position within the defined the volume is observable by plural of the cameras, the computer system can use stereovision calculations to determine the position of any given item or marker that appears in two camera images and can use multivision calculations (as discussed below) to determine the position of any given item or marker that appears in more than two camera images. The machine vision system can thus provide passive and continuous monitoring of the defined volume. This is particularly advantageous when new modules are introduced into the volume for connection to existing modules within the volume, as the system can provide continuous updates on changes in the positions and orientations of the new modules, helping construction personnel to correctly install and connect the new modules.

[0021] The vision system is camera-based and therefore passive, making it safe to use around people. The cameras also allow the system to operate more reliably than laser-based measurement systems under foggy or dusty conditions. Other advantages are that cameras are generally straightforward to install, robust in operation, do not require high precision laser optics, can provide high-resolution measurements with colour information, and are relatively low cost.

[0022] Moreover, the cameras enable high-frequency measurements of multiple points simultaneously, providing large data sets which are amenable to statistical analysis to reduce measurement uncertainties. 008812141

[0023] 3

[0024] More specifically, the first aspect can provide a machine vision system for determining positions and orientations of modules during construction of a modular industrial plant in a defined volume to be occupied by the plant, calibration artefacts being located at predetermined reference positions within the defined volume, the machine vision system including: plural cameras placeable in an array which provides the following conditions: (i) the cameras have overlapping fields of view of respective portions of the defined volume whereby, when the defined volume is empty of modules, each position within the defined the volume is viewable by plural of the cameras and the defined volume is dividable into plural non-overlapping sub-volumes which completely fill the defined volume and in which each sub-volume is within the fields of view of a combination of plural viewing cameras, the combination of viewing cameras of each sub-volume being different from the combination of viewing cameras of each other sub-volume, and (ii) each camera has at least two of the calibration artefacts within its field of view and each calibration artefact is viewable by at least two of the cameras, and a computer system which, as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, is programmed to repeatedly: receive images from the cameras, and determine therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras, for each viewable module the determination being based on a stereovision or multivision calculation which uses the observations of said viewable module within the received images containing said viewable module as well as using the observations of the respective calibration artefacts within the received images containing said viewable module; whereby, under the constraint of conditions (i) and (ii), the repeated determination of the positions and orientations by the computer system results in a given new module being continuously motion tracked by the computer system through the defined volume as it moves on a journey through adjacent sub-volumes from a position of introduction into the defined volume to a final connection position in the defined volume.

[0025] Implementing such a machine vision system with, for example, permanent calibration artefacts, rapid hardwired data transfer, vibration isolation, and high-powered central data processing, can help to ensure that all position and orientation determinations are quickly and accurately mapped to a global positioning value of the defined volume. Such determinations would not be possible with the system proposed in US 2018 / 0343421.

[0026] The industrial plant may be a small nuclear reactor (e.g. having an electrical generation capacity of up to 500 MW), and in this case the defined volume may be within a building of the reactor. However, this does not exclude that the plant may be e.g. another type of power plant, a chemical processing plant, a manufacturing plant etc.

[0027] Conveniently, the array of cameras may be an overhead array of cameras whose fields of view are directed downwards into the defined the volume. For example, the cameras may be attached to a gantry 008812141

[0028] 4 arrangement, e.g. located at ceiling level in a building housing the plant. As the modules are installed in the plant, the fields of view of the overhead array of cameras are gradually filled with modules. Advantageously, however, as the modules are typically installed in the plant from the lowest levels upwards, with subsequent modules being located above and / or sideways relative to previously installed modules, the cameras are still able to view the empty spaces of the defined volume into which further modules will be installed.

[0029] The cameras may be arranged in the array such that within the defined volume, but with increasing distance from the array, positions fall within the fields of view of increasing numbers of cameras. In this way, although camera resolution and hence stereovision positional measurement accuracy achievable by any pair of cameras reduce with increasing distance from the cameras, these reductions can be mitigated at least to an extent by improvement in accuracy provided by introducing more pairs of cameras, and / or camera pairs with greater baseline distance between them, into the calculations (which thus become multivision calculations i.e. being plural, parallel, stereovision calculations). Thus preferably the cameras are arranged such that when the defined volume is empty of modules, each position within the defined volume is within the fields of view of three or more of the cameras.

[0030] The modules may have reference markers at predetermined locations thereon, the references markers being included in the received images and being used by the computer system to determine (e.g. by performing the stereovision or multivision calculations) the positions and orientations of the modules within the defined volume. By utilising such reference markers, the task of the computer system in determining module positions and orientations from the received images can be simplified, and reliability and accuracy improved. In particular, rather than having to recognise characteristics of entire modules in the images, the computer system only has to be able to recognise characteristics of the markers, which can in turn be configured to facilitate this recognition. For example, the reference markers may be located at upper corners of the modules and / or at predefined locations (e.g. midway) along upper edges of the modules, so that they are visible to an overhead camera array and allow the shape of at least an upper face of the module to be determined. Some or all of the reference markers of a give module may be individually distinguishable from the other markers of the module such that the orientation of the module may be determinable therefrom. For example, if markers are located at the four corners of an upper quadrilateral face of the module, one of these markers may be distinguishable by one or more of shape, pattern, colour etc. from the others to allow the rotational orientation that that face presents to the cameras to be determined. Additionally or alternatively, a marker, or a group of markers in combination, may have no rotational symmetry such that the orientation of the module may be determined just from the orientation of that marker or group. Many types and arrangements of reference markers are known to the skilled person which would allow the position and orientation of a given module to be determined by the computer system from the camera images. Well-designed reference markers are high-contrast and promote measurement accuracy. In particular, they can enable effective image deconvolution, reducing uncertainty around the position of, typically, the central point of a marker, thus increasing the accuracy of position measurement based on the marker. 008812141

[0031] 5

[0032] The computer system may be configured to receive a unique identifier of each new module introduced into the defined volume. For example, the modules may have unique identifiers thereon, the unique identifiers being included in the received images and being used by the computer system to identify the modules within the defined volume. These unique identifiers may be in addition to or combined with the aforementioned reference markers. Additionally or alternatively, the computer system may have a human-machine interface through which construction personnel can input the unique identifier of each new module introduced into the defined volume. In whatever way the computer system receives the unique identifiers, the computer system preferably includes a database correlating unique identifiers of the modules and their respective, correct three dimensional positions and orientations in the defined volume. In this case, the computer system may then be further programmed to: compare, for each new module identified and introduced into the defined volume, the new module’s determined position and orientation against the correct position and orientation obtained from the database for that module, and issue guidance as to whether the new module is correctly positioned and oriented or not. For example, the machine vision system may conveniently further include one or more portable terminals which receive the guidance and display it on respective screens of the terminals. Thus the machine vision system can provide construction personnel with interactive assistance in regard of the positioning and orientation of the modules.

[0033] The computer system may be further programmed to repeatedly: scan the images for characteristic identifiers of construction workers and / or construction equipment, and issue notifications when the scanning identifies said characteristic identifiers. For example, the aforementioned portable terminals may receive the notifications and display them on the screens of the terminals. In this way, by raising awareness of the presence of operators within the defined volume, the machine system can improve construction site safety.

[0034] The computer may also be programmed to repeatedly: filter the received images to reduce their information content, e.g. filtering to retain in the images only the reference markers and / or the unique identifiers and / or the characteristic identifiers discussed above. In this way, data storage and processing requirements of the computer system can be reduced.

[0035] In a second aspect, the present invention provides a construction site for constructing a modular industrial plant, the construction site having the machine vision of the first aspect installed therein such that the construction site includes the defined volume.

[0036] The construction site may further have the calibration artefacts located at the predetermined reference positions within the defined volume.

[0037] The construction site may further have one or more connected modules of the modular industrial plant located in the defined volume. 008812141

[0038] 6

[0039] The construction site may further have an overhead gantry arrangement above the defined the volume to which the array of cameras is attached.

[0040] In a third aspect, the present invention provides a method of constructing a modular industrial plant in a defined volume to be occupied by the plant, the method including: providing an array of cameras having overlapping fields of view such that when the defined volume is empty of modules, each position within the defined the volume is observable by plural of the cameras, and as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, repeatedly performing the steps of: receiving images from the cameras, and determining therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras.

[0041] More specifically, the third aspect can provide a method of constructing a modular industrial plant in a defined volume to be occupied by the plant, calibration artefacts being located at predetermined reference positions within the defined volume, the method including: placing plural cameras in an array which provides the following conditions: (i) the cameras have overlapping fields of view of respective portions of the defined volume whereby when the defined volume is empty of modules, each position within the defined the volume is viewable by plural of the cameras and the defined volume is dividable into plural non-overlapping sub-volumes which completely fill the defined volume and in which each sub-volume is within the fields of view of a combination of plural viewing cameras, the combination of viewing cameras of each sub-volume being different from the combination of viewing cameras of each other sub-volume, and (ii) each camera has at least two of the calibration artefacts within its field of view and each calibration artefact is viewable by at least two of the cameras, and as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, repeatedly performing the steps of: receiving images from the cameras, and determining therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras, for each viewable module the determination being based on a stereovision or multivision calculation which uses the observations of said viewable module within the received images containing said viewable module as well as using the observations of the respective calibration artefacts within the received images containing said viewable module; whereby, under the constraint of conditions (i) and (ii), the repeated determination of the positions and orientations by the computer system results in a given new module being continuously motion tracked through the defined volume as it moves on a journey through adjacent sub-volumes from a position of introduction into the defined volume to a final connection position in the defined volume. 008812141

[0042] 1

[0043] The method of the third aspect corresponds to, and may be performed using, the machine vision system of the first aspect. Accordingly, optional features of the machine vision system of the first aspect pertain also to the method of the third aspect.

[0044] For example, the array of cameras may be an overhead array of cameras whose fields of view are directed downwards into the defined the volume.

[0045] For example, the cameras may be arranged in the array such that within the defined volume, but with increasing distance from the array, positions fall within the fields of view of increasing numbers of cameras and / or camera pairs with greater baseline distance between them.

[0046] For example, when the defined volume is empty of modules, the cameras may be arranged such that each position within the defined volume is within the fields of view of three or more of the cameras.

[0047] For example, the modules may have reference markers at predetermined locations thereon, the references markers being included in the received images and being used to determine (e.g. by performing the stereovision or multivision calculations) the positions and orientations of the objects within the defined volume.

[0048] For example, the method may further include receiving a unique identifier of each new module introduced into the defined volume. More particularly, the method may further include: providing a database correlating unique identifiers of the modules and their respective, correct three dimensional positions and orientations in the defined volume; comparing, for each new module identified and introduced into the defined volume, the new module’s determined position and orientation against the correct position and orientation obtained from the database for that module; and issuing guidance as to whether the new module is correctly positioned and oriented or not.

[0049] For example, the method may further include repeatedly performing the steps of: scanning the images for characteristic identifiers of construction personnel and / or construction equipment; and issuing notifications when the scanning identifies said characteristic identifiers.

[0050] For example, the method may further include repeatedly: filtering the received images to reduce their information content, e.g. filtering to retain in the images only the reference markers and / or the unique identifiers and / or the characteristic identifiers discussed above.

[0051] In a fourth aspect, the present invention provides the computer system of the machine vision system of the first aspect.

[0052] For example, this aspect can provide a computer system of a machine vision system for determining positions and orientations of modules during construction of a modular industrial plant in a defined volume to be occupied by the plant, the machine vision system including an array of cameras having overlapping fields of view such that, when the defined volume is empty of modules, each position within the defined 008812141

[0053] 8 the volume is observable by plural of the cameras; wherein the computer system is programmed, as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, to repeatedly: receive images from the cameras, and determine therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras.

[0054] More specifically, the fourth aspect can provide a computer system of a machine vision system for determining positions and orientations of modules during construction of a modular industrial plant in a defined volume to be occupied by the plant, calibration artefacts being located at predetermined reference positions within the defined volume, and the machine vision system including plural cameras placeable in an array which provides the following conditions: (i) the cameras have overlapping fields of view of respective portions of the defined volume whereby, when the defined volume is empty of modules, each position within the defined the volume is viewable by plural of the cameras and the defined volume is dividable into plural non-overlapping sub-volumes which completely fill the defined volume and in which each sub-volume is within the fields of view of a combination of plural viewing cameras, the combination of viewing cameras of each sub-volume being different from the combination of viewing cameras of each other sub-volume, and (ii) each camera has at least two of the calibration artefacts within its field of view and each calibration artefact is viewable by at least two of the cameras; wherein the computer system is programmed, as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, to repeatedly: receive images from the cameras, and determine therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras, for each viewable module the determination being based on a stereovision or multivision calculation which uses the observations of said viewable module within the received images containing said viewable module as well as using the observations of the respective calibration artefacts within the received images containing said viewable module; whereby, under the constraint of conditions (i) and (ii), the repeated determination of the positions and orientations by the computer system results in a given new module being continuously motion tracked by the computer system through the defined volume as it moves on a journey through adjacent sub-volumes from a position of introduction into the defined volume to a final connection position in the defined volume.

[0055] Optional features of the computer system of the machine vision system of the first aspect pertain also to the computer system of the fourth aspect.

[0056] The invention includes the combination of the aspects and preferred features described except where such a combination is clearly impermissible or expressly avoided. 008812141

[0057] 9

[0058] Summary of the Figures

[0059] Embodiments and experiments illustrating the principles of the invention will now be discussed with reference to the accompanying figures in which:

[0060] Figure 1 shows schematically a cross-section through a small modular reactor industrial plant;

[0061] Figure 2 shows one of the modules of the plant;

[0062] Figure 3A shows schematically a machine vision system installed in a building of the plant;

[0063] Figure 3B shows schematically a defined volume viewable by a camera array of the machine vision system of Figure 3A;

[0064] Figure 4 shows schematically a view of the top face of a module; and

[0065] Figure 5 shows a typical workflow for positioning a new module.

[0066] Detailed Description of the Invention

[0067] Aspects and embodiments of the present invention will now be discussed with reference to the accompanying figures. Further aspects and embodiments will be apparent to those skilled in the art. All documents mentioned in this text are incorporated herein by reference.

[0068] Figure 1 shows schematically a cross-section through a small modular reactor industrial plant 1 , and Figure 2 shows just one of the modules 2 of the plant. The modules are housed in buildings 3 of the plant and are generally rectangular cuboid in shape, being formed with a steel frame 4 that defines the edges and corners of the cuboids. Equipment items 5 such as pressure vessels, pipes, pumps, heat exchangers etc. are contained within the modules, which have to be located in the plant buildings and located relative to each other in a precise manner such that suitable connections between the equipment items of adjacent vessels can be made. As previously noted, advantages of constructing plant in this way are that the modules can be factory-fabricated, which reduces costs, lead-times and risk, and increase quality. Typically the modules are installed in the plant from the lowest levels upwards, with subsequent modules being stacked above and / or sideways relative to already-installed modules.

[0069] Figure 3A shows schematically a machine vision system installed in one of the buildings of the plant 1 . The building 3 has a ceiling-level gantry arrangement 6 to which are mounted a downwardly directed array of cameras 7 of a machine vision system. More particularly, the building 3 has a concrete base 8 which supports the gantry arrangement, vibration isolators 9 at the feet of the pillars of the gantry arrangement helping to prevent reduce vibrations in the cameras 7. To further enhance the positional stability of the camera array, the gantry arrangement can be configured to have high stiffness and low thermal expansion. 008812141

[0070] 10

[0071] The camera array typically contains plural parallel rows of cameras 7, Figure 3A showing just one of the rows. The cameras have respective, overlapping fields of view (FOVs), the limits of which are indicated by dashed lines in Figure 3A. In order to make a stereovision measurement of the position of a point using images from the cameras, it is necessary that the point is within the FOVs of at least two cameras. The camera array provides a full site defined volume 10, indicated in Figure 3A by the grey shaded region, with each point in this volume being within the fields of view of at least two cameras. Accordingly, the positions and orientations of modules 2 moved into the defined volume can be determined using images from the cameras. As the modules are installed in the plant, the fields of view of the cameras are gradually filled with modules. However, as the cameras are pointed downwards, they are still able to view the empty spaces above or adjacent to installed modules where further modules will be stacked.

[0072] Figure 3B shows schematically the defined volume 10 viewable by the array of cameras 7 of the machine vision system of Figure 3A. The defined volume 10 can be dividable into plural non-overlapping subvolumes 16 which completely fill the defined volume. Each sub-volume is within the fields of view of a combination of plural viewing cameras, but the combination of viewing cameras of each sub-volume is different from the combination of viewing cameras of each other sub-volume. Additionally, calibration artefacts 17 (e.g. high contrast 3D cube objects with precisely known dimensions and key feature points at their corners) are accurately located at predetermined reference positions within the defined volume 10 to meet the following viewability conditions: each camera has at least two of the artefacts within its field of view and each artefact is viewable by at least two of the cameras. The camera and artefact arrangement can be flexibly optimised depending on site requirements, but typically allows the machine vision system to be calibrated according to a global (full site volume) coordinate system.

[0073] For example, the artefact positions can be selected so that the viewability conditions are maintained even as the defined volume is filled with modules. The calibration setup may include permanent calibration artefacts 17 at a designated origin (0,0,0) point and at least three other designated corners, with other artefacts positioned throughout the viewable volume. Alternatively, if the volume of interest is likely to change throughout use and absolute positioning relative to objects outside the current volume is not required, permanent artefacts may not be required, with a new designated origin point established at the start of each recalibration.

[0074] In addition to the camera array, the machine vision system includes a computer system 13 which receives images from the cameras 7 and performs the stereovision / multivision calculations which result in the determination of positions and orientations, relative to each other, of those modules 3 of the plant within the defined volume which are each viewable by plural of the cameras 7. In overall terms, the cameras 7 capture different perspective images and the computer system computes the relative displacement (known as the disparity) between corresponding points on the modules and calibration artefacts in the acquired images to produce a disparity map. By using triangulation, the obtained disparity map leads to depth information and hence, as the camera positions and the reference positions of the calibration artefacts 17 are known in a coordinate system of the defined volume, determination of module position 008812141

[0075] 11 and orientation in that coordinate system. Significantly, because each sub-volume 16 is within the fields of view of a combination of plural cameras, and because each camera has at least two of the calibration artefacts within its field of view and each calibration artefact is viewable by at least two of the cameras, repeated determination of the module positions and orientations results in a given new module being continuously motion tracked in real time by the computer system through the defined volume as it moves on a journey through adjacent sub-volumes from a position of introduction into the defined volume to a final connection position in the defined volume. More particularly, an unbroken chain of datum points is distributed throughout the defined volume 10, allowing allow all position and orientation measurements to be quickly and precisely mapped to their global position coordinates.

[0076] Assuming array and camera characteristics it is possible to calculate typical achievable resolutions for the vision system. For example, in the case of a square array of cameras with a 10 m spacing between neighbouring cameras where each camera has a focal length of 25 mm, a sensor size of 20 mm x 20 mm, a pixel pitch in the sensor of 2.5 pm, and an angle of view 43.6°, then the overlapping fields of view of any two adjacent pairs of the cameras can provide the following resolutions (assuming a pixel accuracy of 0.2):

[0077] Table 1 : Resolutions for different distances from camera array

[0078] The computer system 13 can implement the stereovision process using calibration techniques known the skilled person. Typically, for example, the system stores intrinsic and extrinsic parameters of each camera and combines these in a camera projection matrix, which is then used to convert the 2D pixel coordinates of camera images into a theoretical outgoing “ray” line, along which the 3D point of an object in the world is known to lie. The intersection of ray lines for a corresponding point from two or more cameras in different locations allows for the 3D point in the world to be fully defined via triangulation. Intrinsic parameters include: focal length, pixel size, principal point coordinates, pixel aspect ratio (typically 1 unless the pixels are non-square) and pixel skew (typically 0 unless the pixels are shaped like rhombi or parallelograms). Extrinsic parameters include: a 3x3 rotation matrix representing the camera’s 008812141

[0079] 12 orientation in space, and a translation vector specifying the camera’s position relative to the world origin, these allowing the transformation between world coordinates to camera coordinates. Further information on stereo system calibration techniques can be found, for example, in R. Beschi et al., Stereo camera system calibration: the need of two sets of parameters, arXiv:2101 .05725v1 , submitted on 14 Jan 2021 , and B. Shan et al., A calibration method for stereovision system based on solid circle target, Measurement, 132 (2019) 213-223.

[0080] The computer system can also perform bundle adjustment. This is a unified method, known to the skilled person, for combining the projection matrix conversions of multiple cameras in a statistically optimal manner. Bundle adjustment refines the parameters of 3D reconstructions from multiple images and camera views with an aim of reducing reprojection errors to improve the accuracy and robustness of 3D models whilst also reducing uncertainty. Typically, it involves an optimization to minimise the reprojection error between observed and predicted image points using nonlinear least-squares algorithms (e.g., Levenberg-Marquardt). Further information on bundle adjustment can be found, for example, in Yu Chen et al., Bundle Adjustment Revisited, arXiv:1912.03858, submitted on 9 Dec 2019.

[0081] As illustrated in Figure 3A, with increasing distance from the camera array, points fall within the fields of view of increasing numbers of cameras. In this way, although camera resolution and hence stereovision positional measurement accuracy achievable by any pair of cameras reduce with increasing distance from the cameras (see Table 1), these reductions can be mitigated at least to an extent by improvement in accuracy provided by introducing more pairs of cameras into multivision calculations, some of these pairs of cameras advantageously also having increased baseline distances therebetween. Thus preferably the cameras are arranged such that, when the defined volume is empty of modules, each position within the defined volume is within the fields of view of three or more of the cameras.

[0082] The vision system effectively performs continuous machine vision measurements, implementing stereovision / multivision in a global coordinate system which enables reliable and accurate measurement and tracking across a large volume at higher resolution than is possible with single camera or conventional stereovision systems. Typically, each camera 7 can obtain and send images to the computer system 13 at rates of about 1 to 100 frames per second, although even higher frame rates are possible. The computer system in turn performs multivision calculations using the received images to form a 3D map of newly introduced module, surrounding installed modules, and the site floor. The position and orientation of each new module relative to the installed modules and site floor are then calculated. The data stereovision / multivision approach is compatible with national and international standards (particularly when using calibration artefacts), as it is a true measurement based on physical principles (i.e. parallax between cameras), and not an inferred measurement based on algorithmic or Al pose / position estimation. Moreover, the data generated by the machine vision system can be subject to statistical manipulation to further improve measurement accuracy.

[0083] To facilitate module recognition in the images received by the computer system 13, each rectangular cuboid module 2 typically includes reference markers at predetermined locations thereon, such as at the 008812141

[0084] 13 corners and / or midpoints of edges one or more faces of the module. In general, if the camera array is ceiling-mounted, it is only necessary to provide the refence markers on the top face, as just this face will be visible to the cameras. Figure 4 shows schematically a view of the top face of a module 2 defined by its steel frame 4, cross-shaped references markers 11 , 11 ’ being located at the corners and midpoints of edges of the face. The computer system can be programmed to perform image filtering prior to calculating the disparity map in order to recognise specifically this type of marker and to ignore other features in the received images. This can increase processing speeds by significantly reducing the amount of data the system needs analyse in order to determine the position and orientation of the module relative to other modules in the defined space. The reliability and accuracy of the determination can also be improved. As shown in Figure 4, one of the references markers 11 ’ is distinguishable by its colour from the other reference markers 11. By differentiating the references markers on the basis of their colour or other attribute (e.g. shape, size etc.), the computer program can thus determine not only the position of the module relative to the positions of other modules in the defined space, but also its orientation. As would be known to the skilled person, however, there are numerous other ways by which reference markers can be used to determine orientation. For example, a reference marker simply without rotational symmetry can be used as an indicator of orientation. Further information on image filtering can be found, for example, in Jiangbo Lu et al., PatchMatch Filter: Efficient Edge-Aware Filtering Meets Randomized Search for Fast Correspondence Field Estimation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 1854-1861.

[0085] In addition to determining position and orientation, the machine vision system typically also receives a unique identifier for each new module 2 introduced into the defined volume. When the identifier is affixed to the module (e.g. as a part of one or more of the previously-mentioned reference markers or as a separate i.d. marker) the system can determine this identifier automatically, for example by performing image filtering to recognise the identifier in the received images. In Figure 4, an i.d. marker 12 is affixed to the centre of the top face of the module for reading by the vision system, in this case the unique identifier being “A34”. Alternatively or additionally a human-machine interface can be provided through which construction personnel can input the unique identifier of each new module introduced into the defined volume. The computer system 13 preferably also includes a database 14 correlating the unique identifier of each module with the, 3D position coordinates and orientation for that module in the defined volume when the module is correctly installed. The database may also contain further information, such as the name and / or function of module, the equipment it contains, and how it is to be connected to other modules. Advantageously, the database allows the computer system not only to monitor the position and orientation of a module as it is moved through the defined volume, but also allows it to issue guidance as to whether the new module is correctly positioned and oriented or not. For example, construction personnel can be provided with wireless-enabled portable terminals 15 which are continuously updated with live module position information transmitted by the computer system so that they can immediately understand from the terminals where newly introduced modules are in relation to existing modules, and also know if a new module has arrived at its correct final location. 008812141

[0086] 14

[0087] The information provided to construction personnel can be enhanced with additional safety information. In particular, the computer system 13 can be configured to identify (e.g. by image filtering) and track characteristic identifiers of construction workers and / or construction equipment (e.g. hard hats, vehicle markers etc.) and present this safety information as notifications to other construction personnel, e.g. on the aforementioned terminals 15. As a specific example, the machine vision system can be used to control swarm cranes on the site. For example, encoders in the swarm cranes, or the beams along which they move, can allow the position of each crane to be integrated into the global coordinate system of the defined volume 10. Furthermore, like the modules 2, reference markers can be attached to e.g. the end of each crane’s hoist so that the position of the hoist can be tracked by the machine vision system. Such adaptations help to improve safety and allow control of crane body and hoist position to be integrated with the machine vision system.

[0088] A typical workflow for positioning a new module is illustrated in Figure 5. At step S1 a new module is brought onto site in the defined volume where it can be imaged by the camera array and moved into approximate position by an installation team. Next, at step S2 the computer system issues determines whether the position and orientation of the module are within specification. If no, it issues guidance (step S3) that the module must be repositioned, and at step S4 the team perform that re-positioning and the workflow then loops back to step S2. Alternatively, if the determination by computer system issues at step S2 is yes, the installation team can move on at step S5 to fixing the module in position. After fixing, at step S6 a further determination is made by the computer system as to whether the position and orientation of the module are still within specification. If no, at step S7 the computer system issues guidance that the module must be unfixed and re-positioned, and at subsequent step S8 the team perform that re-positioning and the workflow then loops back to step S2. Alternatively, if the determination by computer system issues at step S6 is yes, the positioning and fixing procedure is complete and the team can move on at step S9 to operatively connecting the module to the adjacent modules.

[0089] The features disclosed in the foregoing description, or in the following claims, or in the accompanying drawings, expressed in their specific forms or in terms of a means for performing the disclosed function, or a method or process for obtaining the disclosed results, as appropriate, may, separately, or in any combination of such features, be utilised for realising the invention in diverse forms thereof.

[0090] While the invention has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the invention set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may be made without departing from the spirit and scope of the invention. 008812141

[0091] 15

[0092] For the avoidance of any doubt, any theoretical explanations provided herein are provided for the purposes of improving the understanding of a reader. The inventors do not wish to be bound by any of these theoretical explanations.

[0093] Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described.

[0094] Throughout this specification, including the claims which follow, unless the context requires otherwise, the word “comprise” and “include”, and variations such as “comprises”, “comprising”, and “including” will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0095] It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” one particular value, and / or to “about” another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent “about,” it will be understood that the particular value forms another embodiment. The term “about” in relation to a numerical value is optional and means for example + / - 10%.

[0096] References

[0097] A number of publications are cited above in order to more fully describe and disclose the invention and the state of the art to which the invention pertains. Citations for these references are provided below. The entirety of each of these references is incorporated herein.

[0098] US 2018 / 0343421.

[0099] R. Beschi et al., Stereo camera system calibration: the need of two sets of parameters, arXiv:2101 .05725v1 , submitted on 14 Jan 2021 .

[0100] B. Shan et al., A calibration method for stereovision system based on solid circle target, Measurement, 132 (2019) 213-223.

[0101] Yu Chen et al., Bundle Adjustment Revisited, arXiv:1912.03858, submitted on 9 Dec 2019.

[0102] Jiangbo Lu et al., PatchMatch Filter: Efficient Edge-Aware Filtering Meets Randomized Search for Fast Correspondence Field Estimation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 1854-1861.

Claims

00881214116Claims:1 . A machine vision system for determining positions and orientations of modules (2) during construction of a modular industrial plant (1) in a defined volume (10) to be occupied by the plant, calibration artefacts (17) being located at predetermined reference positions within the defined volume, the machine vision system including: plural cameras (7) placeable in an array which provides the following conditions: (i) the cameras have overlapping fields of view of respective portions of the defined volume whereby, when the defined volume is empty of modules, each position within the defined the volume is viewable by plural of the cameras and the defined volume is dividable into plural non-overlapping sub-volumes (16) which completely fill the defined volume and in which each sub-volume is within the fields of view of a combination of plural viewing cameras, the combination of viewing cameras of each sub-volume being different from the combination of viewing cameras of each other sub-volume, and (ii) each camera has at least two of the calibration artefacts within its field of view and each calibration artefact is viewable by at least two of the cameras, and a computer system (13) which, as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, is programmed to repeatedly: receive images from the cameras, and determine therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras, for each viewable module the determination being based on a stereovision or multivision calculation which uses the observations of said viewable module within the received images containing said viewable module as well as using the observations of the respective calibration artefacts within the received images containing said viewable module; whereby, under the constraint of conditions (i) and (ii), the repeated determination of the positions and orientations by the computer system results in a given new module being continuously motion tracked by the computer system through the defined volume as it moves on a journey through adjacent sub-volumes from a position of introduction into the defined volume to a final connection position in the defined volume.

2. The machine vision system of claim 1 , wherein the array of cameras is an overhead array of cameras whose fields of view are directed downwards into the defined the volume.

3. The machine vision system of claim 1 or 2, wherein the cameras are arranged in the array such that within the defined volume, but with increasing distance from the array, positions fall within the fields of view of increasing numbers of cameras.

4. The machine vision system of any one of the previous claims, wherein the cameras are arranged such that, when the defined volume is empty of modules, each position within the defined volume is within the fields of view of three or more of the cameras.008812141175. The machine vision system of any one of the previous claims, wherein the modules have reference markers (11 , 11 ’) at predetermined locations thereon, the references markers being included in the received images and being used by the computer system to perform the stereovision or multivision calculations which determine the positions and orientations of the modules within the defined volume.

6. The machine vision system of any one of the previous claims, wherein the computer system is configured to receive a unique identifier (12) of each new module introduced into the defined volume.

7. The machine vision system of claim 6, wherein the computer system includes a database (14) correlating unique identifiers of the modules and their respective, correct three dimensional positions and orientations in the defined volume, wherein the computer system is further programmed to: compare, for each new module identified and introduced into the defined volume, the new module’s determined position and orientation against the correct position and orientation obtained from the database for that module, and issue guidance as to whether the new module is correctly positioned and oriented or not.

8. The machine vision system of claim 7, further including one or more portable terminals (15) which receive the guidance and display it on respective screens of the terminals.

9. The machine vision system of any one of the previous claims, wherein the computer system is further programmed to repeatedly: scan the images for characteristic identifiers of construction workers and / or construction equipment, and issue notifications when the scanning identifies said characteristic identifiers.

10. The machine vision system of claim 9 as dependent on claim 8, wherein the one or more portable terminals receive the notifications and display them on the screens of the terminals.

11. The machine vision system of any one of the previous claims, wherein the computer system is further programmed to repeatedly: filter the received images to reduce their information content.

12. A construction site for constructing a modular industrial plant, the construction site having the machine vision of any one of the previous claims installed therein such that the construction site includes the defined volume, and further having the calibration artefacts located at the predetermined reference positions within the defined volume.

13. The construction site according to claim 12 including an overhead gantry arrangement (6) above the defined the volume to which the array of cameras is attached.

14. A method of constructing a modular industrial plant (1) in a defined volume (10) to be occupied by the plant, calibration artefacts (17) being located at predetermined reference positions within the defined volume, the method including: placing plural cameras (7) in an array which provides the following conditions: (i) the cameras00881214118 have overlapping fields of view of respective portions of the defined volume whereby when the defined volume is empty of modules (2), each position within the defined the volume is viewable by plural of the cameras and the defined volume is dividable into plural non-overlapping sub-volumes (16) which completely fill the defined volume and in which each sub-volume is within the fields of view of a combination of plural viewing cameras, the combination of viewing cameras of each sub-volume being different from the combination of viewing cameras of each other sub-volume, and (ii) each camera has at least two of the calibration artefacts within its field of view and each calibration artefact is viewable by at least two of the cameras, and as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, repeatedly performing the steps of: receiving images from the cameras, and determining therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras, for each viewable module the determination being based on a stereovision or multivision calculation which uses the observations of said viewable module within the received images containing said viewable module as well as using the observations of the respective calibration artefacts within the received images containing said viewable module; whereby, under the constraint of conditions (i) and (ii), the repeated determination of the positions and orientations by the computer system results in a given new module being continuously motion tracked through the defined volume as it moves on a journey through adjacent sub-volumes from a position of introduction into the defined volume to a final connection position in the defined volume.

15. A computer system of a machine vision system for determining positions and orientations of modules (2) during construction of a modular industrial plant (1) in a defined volume (10) to be occupied by the plant, calibration artefacts (17) being located at predetermined reference positions within the defined volume, and the machine vision system including plural cameras (7) placeable in an array which provides the following conditions: (i) the cameras have overlapping fields of view of respective portions of the defined volume whereby, when the defined volume is empty of modules, each position within the defined the volume is viewable by plural of the cameras and the defined volume is dividable into plural non-overlapping sub-volumes (16) which completely fill the defined volume and in which each sub-volume is within the fields of view of a combination of plural viewing cameras, the combination of viewing cameras of each sub-volume being different from the combination of viewing cameras of each other subvolume, and (ii) each camera has at least two of the calibration artefacts within its field of view and each calibration artefact is viewable by at least two of the cameras; wherein the computer system (13) is programmed, as the industrial plant is constructed by introducing new modules into the defined volume and connecting the new modules to modules already located in the defined volume, to repeatedly: receive images from the cameras, and determine therefrom positions and orientations, relative to each other, of those modules of the plant within the defined volume which are each viewable by plural of the cameras, for each viewable module the determination being based on a stereovision or multivision calculation which uses the observations of said viewable module within the received images containing said viewable module as well as using the observations of the00881214119 respective calibration artefacts within the received images containing said viewable module; whereby, under the constraint of conditions (i) and (ii), the repeated determination of the positions and orientations by the computer system results in a given new module being continuously motion tracked by the computer system through the defined volume as it moves on a journey through adjacent sub-volumes from a position of introduction into the defined volume to a final connection position in the defined volume.

Images