A method for automatic image painting on a surface and a painting system

Abstract

The present invention relates to a computer-implemented method (100) for controlling painting of a surface (50) of a three-dimensional (3D) object (2) using a robot system (1), the robot system comprising a nozzle arrangement (14) with a set of controllable nozzles (16) for depositing paint on the surface, wherein the method comprises: providing (110) a digital model (20) containing data of a surface geometry of a virtual surface (50'), the virtual surface representing a virtual counterpart of the surface; positioning (130), in the digital model, a digital image (22a) on the virtual surface (50'), the digital image representing an image (22b) to be painted on the surface; determining (140), in the digital model, one or more nominal reference points (24a to 24n) associated with the positioned digital image on the virtual surface, the nominal reference points indicating zones (25a to 25n) potentially affecting the visual representation of the digital image; determining (150) an actual painting surface geometry (27); determining (160) locations of one or more actual reference points (24a1 to 24n1) of the zones (25a1 to 25n1) on the surface; determining (170) a deviation; adapting (180) the digital image; and controlling (190) the robot system to paint on the surface according to the adapted digital image.

Description

[0001] A METHOD FOR AUTOMATIC IMAGE PAINTING ON A SURFACE AND A PAINTING SYSTEM

[0002] Technical Field

[0003] The present invention relates to the field of automated painting by means of a robot-carried paint nozzle. By way of example, the present invention relates to methods, controllers and robot systems for painting and controlling painting of one or more images on a surface of a three-dimensional (3D) object. In particular, the present invention relates to methods and systems for automatic image painting on an external surface of a vehicle or the like.

[0004] Background

[0005] The painting of complex three-dimensional (3D) surfaces, such as the exterior of vehicles like cars, airplanes, and underwater vessels, may often require precise application of paint to achieve desired aesthetic and functional outcomes. Traditional painting methods often involve manual intervention and can be time-consuming and prone to human error. With the advent of robotic systems, there has been a push towards automating the painting process.

[0006] Despite the progress in the field, there remains a need for improvements in further automating the process so as to allow for enhanced painting of images, such as decals, on interior and exterior surface of 3D objects, such as interior and exterior surfaces on vehicles.

[0007] Summary

[0008] In view of the above, it would be desirable to provide an improved method for painting one or more images on the surface of a three-dimensional (3D) object, where the paint and images are both applied in a more accurate and automated manner. The object is at least partly achieved by a computer- implemented method according to independent claim i, and the present invention as defined in the other independent claims. The dependent claims relate to advantageous embodiments.

[0009] According to a first aspect of the present invention, there is provided a computer-implemented method for controlling painting on a surface of a three-dimensional 3D object using a robot system. The robot system comprises a nozzle arrangement with a set of controllable nozzles for depositing paint on the surface. The method comprises providing a digital model containing data of a surface geometry of a virtual surface, the virtual surface representing a virtual counterpart of the surface; positioning, in the digital model, a digital image on the virtual surface, the digital image representing an image to be painted on the surface; determining, in the digital model, one or more nominal reference points associated with the positioned digital image on the virtual surface, the nominal reference points indicating zones potentially affecting the visual representation of the digital image; determining an actual painting surface geometry of the surface from measured geometry data of the surface; based on the determined actual painting surface geometry, determining locations of one or more actual reference points of the zones on the surface; determining a deviation between the determined one or more nominal reference points and the determined locations of the one or more actual reference points; adapting the digital image in response to the determined deviation; and controlling the robot system to paint on the surface according to the adapted digital image.

[0010] The computer-implemented method is typically performed by a controller having a processing circuitry. Hereby, there is provided an improved method for painting one or more images on the surface of a three-dimensional (3D) object, where the application of paint and images may be performed with greater accuracy and automation. The proposed method addresses challenges associated with aligning digital images to the actual surface of 3D objects, thereby reducing visible defects and minimizing the need for manual adjustments during the printing process.

[0011] More specifically, the invention is at least partly based on the insight that current robotic painting systems still face difficulties in accurately aligning digital images with the actual surfaces of 3D objects due to geometric deviations between the nominal CAD model and the real-world object. That is, an image intended to be painted on the surface may be configured to align precisely with the nominal CAD model; however, the image may not correspond exactly to the physical object after painting, due to inherent manufacturing deviations or tolerances in the dimensions of the physical object. The method involves dynamically modifying the digital image based on real-time measurements of key geometric reference points on the actual object, thereby allowing for more precise alignment with the object’s contours and features.

[0012] By determining and comparing nominal and actual reference points on the object’s surface, the method permits adjustments to the digital image based on the measured deviations between these points. Such adjustments may help the image align more closely with the actual contours and features of the object, reducing the likelihood of visible mismatches that can result from differences between the nominal CAD model and the real object surface.

[0013] The method includes an automatic adjustment of the digital image in response to measured geometry data, which may reduce the need for manual reprogramming or regeneration of the image data. Allowing for corrections during the production process may potentially save time and reduce operational complexity. Incorporating the determination of the actual painting surface geometry from measured geometry data and adapting the painting instructions accordingly further supports handling variations in object geometry. Such capability may be particularly useful in industrial settings where objects do not perfectly match their nominal CAD designs, allowing for more consistent quality across different production lots.

[0014] Moreover, the proposed method enables modification of the image based on real-time measurements, providing additional versatility for different printing tasks. Whether applying complex patterns or simple two-tone designs, the method can adjust the image to fit the specific contours and dimensions of each object. Flexibility in application may make the method suitable for a wide range of industrial uses, from detailed graphics to large surface color applications.

[0015] To this end, the proposed method allows for a more precise, efficient, and adaptable process of applying images to 3D surfaces, contributing to improved overall performance in automated painting systems.

[0016] The surface may typically be any one of an interior and exterior surface of 3D object, such as a vehicle, e.g. a car, an airplane, underwater vessel and the like. The surface defines a part of a surface of the 3D object, or the complete surface of the 3D object.

[0017] The provision of determining an actual painting surface geometry of the surface may be performed by a spatial data acquisition device. A technical advantage may include enhanced accuracy in capturing the actual surface geometry of the surface, as the use of a spatial data acquisition device like a 3D scanner allows for precise and detailed measurement, leading to more accurate alignment of the digital image during the painting process.

[0018] The provision of determining a deviation between the determined one or more nominal reference points and the determined locations of the one or more actual reference points may be performed using a best fit algorithm. A technical advantage may include improved precision in calculating deviations between nominal and actual reference points, as the use of a best fit algorithm can optimize the alignment process, resulting in more accurate adaptation of the digital image to the actual surface.

[0019] The provision of adapting the digital image in response to the determined deviation comprises aligning the position of the digital image in the digital model relative to the determined actual painting surface geometry of the surface. A technical advantage may include more accurate positioning of the image on the surface. By aligning the digital image based on the actual surface geometry, the method may reduce the likelihood of misalignment and enhancing the overall quality of the painted image.

[0020] The provision of aligning the position of the digital image in the digital model relative to the determined actual painting surface geometry of the surface may comprise adding or removing one or more pixels in the digital image. A technical advantage may include increased flexibility and precision in image adaptation, as adding or removing pixels allows the digital image to be finely tuned to match the actual surface profile, resulting in a more accurate and visually consistent painted image.

[0021] The provision of adapting the digital image in response to the determined deviation comprises aligning the digital image on the surface in discrete segments, each segment corresponding to one or more actual reference points on the surface, and adjusting the dimensions of the digital image within each segment to compensate for deviations in the positions of the reference points relative to the actual painting surface geometry. A technical advantage may include improved adaptability and precision in image alignment, as the method allows for segmenting the image and making localized adjustments. This segmentation can better accommodate complex surface geometries, ensuring that each segment of the image accurately conforms to the corresponding part of the surface.

[0022] The zones on the surface may correspond to locations with distinct geometric features. A technical advantage may include enhanced accuracy in image placement, as aligning the image based on zones with distinct geometric features allows for more precise correspondence between the image and the physical characteristics of the surface, leading to improved visual outcomes.

[0023] According to a second aspect of the invention, there is provided a controller comprising processing circuitry configured to execute the method of the first aspect of the invention. The second aspect of the invention may seek to solve the same problem as described for the first aspect of the invention. Thus, effects and features of the second aspect of the invention are largely analogous to those described above in connection with the first aspect of the invention. Another technical advantage may include greater integration and efficiency in executing the method, as a controller with dedicated processing circuitry ensures the method’s steps are carried out effectively, leading to reliable and consistent performance in painting applications.

[0024] According to a third aspect of the invention, there is provided a robot system, comprising: a nozzle arrangement configured to deposit paint onto a surface; a robot arm arrangement configured to move the nozzle arrangement over the surface; and a controller comprising processing circuitry configured to execute the method of the first aspect of the invention. The third aspect of the invention may seek to solve the same problem as described for the first and second aspects of the invention. Thus, effects and features of the third aspect of the invention are largely analogous to those described above in connection with the first and second aspects of the invention. An additional technical advantage may include enhanced automation and precision in the painting process, as the robot system is equipped with a controller that can execute the method.

[0025] The robot system may be an industrial robot system and the controller may be an integral part of a robot control system configured to control the robot arm arrangement and the nozzle arrangement. Such system may reduce the complexity of the overall system architecture, while ensuring synchronized operation of the robot arm arrangement and nozzle arrangement. The controller may be configured to determine the actual painting surface geometry of the surface using a spatial data acquisition device. The controller may be configured to determine a deviation between the one or more nominal reference points and the one or more actual reference points using a best-fit algorithm. The controller may be configured to adapt the digital image in response to the determined deviation by aligning the position of the digital image in the digital model relative to the determined actual painting surface geometry of the surface. The controller may be configured to align the position of the digital image in the digital model by adding or removing one or more pixels in the digital image.

[0026] There is also provided a computer program product comprising program code for performing, when executed by a controller, the method according to the first aspect, and a non-transitory computer-readable storage medium comprising instructions, which when executed by a controller, cause the controller to perform the method according to the first aspect. The computer program may be stored or distributed on a data carrier. As used herein, a “data carrier” may be a transitory data carrier, such as modulated electromagnetic or optical waves, or a non-transitory data carrier. Non- transitory data carriers include volatile and non-volatile memories, such as permanent and non-permanent storage media of magnetic, optical or solid- state type. Still within the scope of “data carrier”, such memories may be fixedly mounted or portable.

[0027] Further features of, and advantages with, the present invention will become apparent when studying the appended claims and the following description. The skilled person realize that different features of the present invention may be combined to create embodiments other than those described in the following, without departing from the scope of the present invention.

[0028] Brief Description of the Drawings

[0029] These and other aspects of the present invention will now be described in more detail, with reference to the appended drawings showing example embodiments of the invention, wherein:

[0030] Fig. i schematically illustrates an example of a robot system according to the present invention;

[0031] Fig. 2 is a flow-chart of an example of a method for painting of one or more images on a surface of a 3D object, in which the painting is controlled and performed by the robot system of Fig. 1, according to the present invention;

[0032] Fig. 3 schematically illustrates an example of further details of the method, in which an alignment of a digital image is adapted in response to a deviation between one or more nominal reference points and one or more actual reference points, according to the present invention; and

[0033] Fig. 4 schematically illustrates an example of determining an actual painting surface geometry of the surface of the 3D object, according to the present invention.

[0034] Detailed Description

[0035] In the present detailed description, various embodiments of robot systems and methods are mainly described with reference to a robot system comprising an industrial robot. However, the described robot system is suitable for any type of system and arrangement comprising a robot, such as an industrial robot, a service robot and the like. Moreover, the described method, control system and controller may be suitable for remote operations in relation to the robot system. The same or similar reference numerals will be used to denote the same or similar structural features. For ease of reference, the examples are described in relation to a three-dimensional (3D) object in the form of the vehicle. Turning now to Fig. 1, which schematically represents a first perspective side view of an exemplary robot system 1. In this example, the robot system 1 is an industrial robot system. The robot system i comprises at least one robot manipulator arrangement io. In some robot systems 1, the robot system 1 may comprise a plurality of robot manipulator arrangements io. The robot system i is here integrated into an automatic assembly line for vehicles 2, such as passenger cars. The robot system i is here also configured to perform painting of one or more parts and surfaces of the vehicle 2, as will be further described herein with reference to figures i to 4. As such, the robot system 1 is configured as an automated painting robot system.

[0036] The vehicle 2 is one example of a 3D object having interior and exterior surfaces 50. In Fig. 1, the surface 50 is an exterior surface in the form of a roof surface of the vehicle 2. The surface 50 may typically be an exterior surface of a vehicle 2, e.g. a car, an airplane, vessel, underwater vessel and the like. The surface 50 can extend over the entire part or component, or as illustrated in Fig. 1, over a defined region of a part or component, such as the roof surface of the vehicle 2. In other examples, the surface 50 may likewise be an interior surface of the 3D object, such as an interior surface of the vehicle 2. The automated painting robot system is adapted (e.g., as regards paint-cell size, arm dimensions) for industrial-scale painting, with an ability to paint one or more surfaces 50, where each surface has a spatial extent of at least 0.1 m, such as at least 0.5 m, such as at least 1.0 m, such as at least several meters, such as at least 10 m.

[0037] With reference to Fig. 1, the robot manipulator arrangement 10 is arranged on a base 40, such as a frame. The base 40 is the foundation on which the robot manipulator arrangement 10 is mounted. The base 40 provides stability and is typically designed to allow for easy integration into existing production lines. The base 40 maybe an integral part of the robot manipulator arrangement 10 or a separate part of the robot system 1.

[0038] The robot manipulator arrangement 10 of Fig. 1 comprises a robot arm arrangement 12. For example, the robot arm arrangement 12 is rotatably mounted on the base 40. The robot arm arrangement 12 comprises one or more robot arms 12a to 12c. In Fig. 1, the robot arm arrangement 12 comprises three robot arms, i.e. a first robot arm 12a, a second robot arm 12b, and a third robot arm 12c. The robot arms 12a, 12b, 12c of the robot arm arrangement 12 are mechanical units configured to replicate the motions of a human arm, allowing for a wide range of movements. Thus, each one of the robot arms 12a, 12b, 12c typically comprises one or more joints. As schematically depicted in Fig. 1, the robot arms 12a, 12b, 12c are articulated connected to each other via the joint(s). Moreover, the first robot arm 12a is rotatably arranged on the base 40. The joints enable a full 360-degree range of motion, allowing the robot arms 12a, 12b, 12c to rotate, pivot, and move in multiple directions. Such articulation provides the robotic system 1 with flexibility and precision, allowing for complex manipulations in confined spaces. This level of articulation is particularly advantageous in surface treatment applications such as painting. The ability to maneuver seamlessly over complex surfaces provides even and consistent application, making the robot system 1 suitable for tasks that require high-quality finishes, such as automotive painting.

[0039] Moreover, as depicted in Fig. 1, the robot manipulator arrangement 10 comprises a painting head 11. In this example, the painting head 11 is configured to be pivotably connectable to the robot arm 12c of the robot arm arrangement 12. The painting head 11 is here connected to the robot arm arrangement 12 via a connection interface 13 disposed on the robot arm 12c. Hence, at least one of the robot arms comprises the connection interface 13 configured to connect the painting head 11 to the robot arm arrangement 12, as may be gleaned from e.g. Fig. 1. The painting head 11 is here the end effector of the robot manipulator arrangement 10. To this end, the robot system is configured to move the painting head 11 over the various surfaces of the vehicle 2 through the robot arm arrangement 12.

[0040] Turning again to Fig. 1, the robot system 1 comprises a nozzle arrangement

[0041] 14. As shown in Fig. 1, the painting head 11 comprises the nozzle arrangement

[0042] 14. As such, the nozzle arrangement 14 is an integral part of the painting head ii, which is configured to be pivotably connected to the robot arm 12c. Through this configuration, the nozzle arrangement 14 is configured to move over the surface 50 in various directions by manipulation of the robot arm arrangement 12.

[0043] The paint 30 can be provided and supplied in several different ways. Typically, the paint 30 is a colorant. One type of colorant is ink. The paint 30 may also be a coating material or any other type of surface deposition material.

[0044] The nozzle arrangement 14 is here adapted for inkjet printing. One type of inkjet printing is so-called pixel printing, Inkjet printing in the form of pixel printing includes a two-dimensional matrix of individually controllable nozzles 16. Hence, the nozzle arrangement 14 here comprises multiple nozzles 16. Each nozzle 16 is configured to be controllable between an open state and a closed state, in which paint flows out from an outlet of the nozzle 16 upon a firing command from a controller. That is, the nozzle 16 can be controlled in a firing state and in a non-firing state. As such, each nozzle 16 is configured to deposit paint droplets 30. The paint 30 is supplied to the painting head 11 via a paint supply conduit arrangement 31, as shown in Fig 1. The paint supply conduit arrangement 31 is fluidly connected to the nozzle arrangement 14 and the nozzles 16, as illustrated in Fig. 1. The paint supply conduit arrangement 31 is an integral part of the painting head 11 and the robot system 1. The paint supply conduit arrangement 31 is fluidly connected to a paint fluid reservoir (not shown), such as an ink fluid storage. The paint fluid reservoir can be arranged on the robot system 1 or external of the robot system 1.

[0045] It should be noted that the term “colorant” typically encompasses any substance used to impart color, including paint, inks, dyes, pigments, and other material. In implementations adapted for color image printing, the paint supply conduit arrangement 31 may include multiple paint supply lines corresponding to different basic colors of paint. The paint supply lines may be operationally independent, or they may share certain hardware components or control functionalities. Conversely, it is envisioned that one paint supply line maybe at the service of multiple paint nozzles.

[0046] As shown in e.g. Fig. 1, the painting head n with the nozzle arrangement 14 is adapted for movement in multiple directions indicated by the arrows. Typically, the nozzle arrangement 14 through the movement of the painting head 11 is adapted to sweep a painting stroke that is y units wide. In practice, the width y is usually somewhat less than the outer dimensions of the nozzle arrangement 14. In a representative inkjet-printing use case, the number of individually controllable nozzles 16 may be of the order of one thousand, and the width y of the painting stroke may be of the order of 0.1 m. The paint density on the surface 50 is generally determined by the ratio of the number of firing nozzles 16 corresponding to the number of outlets of the nozzles and the speed of movement of the painting head 11 (corresponding to the speed of the nozzle arrangement 14) relative to the surface 50. Controlling the nozzles 16 and paint density, including selecting color etc., can be performed in several different manners depending on type of image and type of surface.

[0047] One example of a painting head 11 is an ink-jet painting head. As such, the nozzle arrangement 14 is configured to deposit ink 30. The deposited ink 30 forms an image on the surface 50. For ease of reference, the painted image on the surface 50 is here indicated by reference numeral 22b.

[0048] Moreover, the robot system 1 comprises a spatial data acquisition device 18, as illustrated in Fig. 1. The spatial data acquisition device 18 is configured to measure spatial data of the surface 50. More specifically, the spatial data acquisition device 18 is configured to measure the surface 50, and identify surface features in the measured geometry data. As such, the measured spatial data contains spatial data of the actual surface 50. By way of example, the spatial data acquisition device 18 is here a scanner, such as a 3D scanner or a 2D line scanner. As illustrated in Fig. 1, the spatial data acquisition device 18 is an integral part of the painting head 11. In other examples, the spatial data acquisition device 18 is arranged at another location of the robot system 1. The spatial data acquisition device 18 is configured to be moved over the surface 50 through the robot arm arrangement 12 in a similar manner as the nozzle arrangement 14. It should be noted that the spatial data acquisition device 18 may in some examples be an integral part of another robot manipulator arrangement.

[0049] In summary, a robot system 1 is provided, comprising the nozzle arrangement 14 with the set of nozzles 16 for depositing paint 30 on the surface 50, the spatial data acquisition device 18, and further comprising the robot arm arrangement 12 configured to move the nozzle arrangement 14 and the spatial data acquisition device 18 over the surface 50.

[0050] As illustrated in Fig. 1, the robot system 1 further comprises a robot controller 90a. The robot controller 90a is configured to control the operation(s) of the robot system 1, including the operations of the robot arm arrangement 12, the operations of the painting head 11 and the operations of the nozzle arrangement 14, as described herein. The robot controller 90a is here an integral part of a control system 90, typically comprises processing circuitry 92. The robot controller 90a is configured to execute one or more control algorithms and motion instructions, thereby managing the robot's movements and operations. In Fig. 1, where the robot system 1 is an industrial robot system, the robot controller 90a is an integral part of a robot control system 90 configured to control the robot arm arrangement 12 and the nozzle arrangement 14. Typically, although strictly not required, the control system 90 comprises one or more controllers 90a, 90b. In such configuration, a first controller 90a maybe arranged in the robot control system 90, and a second controller 90b maybe arranged in the painting head 11, and configured to control the operation of the painting head 11 and / or the nozzle arrangement 14 based on instructions from the first controller 90a. The second controller 90b may thus be in communication with the first controller 90a. However, in other arrangements, the robot control system 90 may comprise a single controller, in which the processing circuitry 92 is configured to control the nozzle arrangement 14 and / or the painting head 11 directly via one or more actuators. Hence, the painting head 11, including the nozzle arrangement 14, can be controlled in in several different manners by various actuators and controllers.

[0051] The control system 90, through the controllers 90a, 90b of the robot system 1 is configured to manage the operations of the nozzle arrangement 14.

[0052] Moreover, as mentioned above, the nozzle arrangement 14 is configured to deposit paint 30 onto the surface 50 of the vehicle 2. As such, in this example, the control system 90 is configured to control the nozzle arrangement 14 over the surface 50 to deposit the paint 30 onto the surface 50. Through the arrangement of the nozzle arrangement 14 in the painting head 11, the control system 90 can control the painting head 11 to deposit the paint 30 onto the surface 50 to form the painted image 22b on the surface 50.

[0053] More precisely, the controller 90a may include processing circuitry 92 which is configured to input a painting sequence for the image, which is stored in e.g. a memory 94 of the control system 90. The painting sequence generates a one or more control signals and commands to the controllers and processing circuitry, whereby a corresponding image is formed on the surface 50. While the painting sequence is here typically generated in a common controller of the control system 90 configured to both generate the painting sequence and control the robot system, as described herein, other options and combinations of generating the painting sequence and controlling the robot system 1 by one or more controllers, one or more processing circuitries and one or more memories may likewise be conceivable.

[0054] The preparation of the painting and the painting on the surface 50 will now be described in relation to Figs. 2 to 4. The preparation of the painting and the painting on the surface 50 is performed by the controller 90a. As such, the control system 90 is here configured to implement a method 100 according to examples. More specifically, the controller 90a comprises processing circuitry 92 configured to perform the method 100 according to the examples. Accordingly, the method 100 is here a computer-implemented method. Fig. 2 is a flowchart of exemplary steps of one example of the method 100. Fig. 3 schematically illustrates the steps of the method 100, in which an alignment of the digital image 22a as applied to a virtual surface 50’ is adapted to the actual surface 50 of the vehicle 50. The method 100 is intended for controlling painting of the image 22b on the surface 50 using the robot system 1 in Fig. 1. The processing circuitry 92 is configured to perform the following steps.

[0055] The method 100 comprises a step of providing no a digital model 20. The digital model 20 contains data of a surface geometry of a virtual surface 50’. The virtual surface 50’ represents a virtual counterpart of the surface 50. As such, while the digital model 20 is typically derived from data related to the actual surface 50, the virtual surface 50’ in the digital model 20 represents the virtual version of the surface 50. In other words, the surface geometry defines the surface profile of the surface 50, as represented in the digital model 20. The digital model 20 here refers to a computer-aided design (CAD) model, such as a CAD geometry model. The digital model 20 may also contain data of the surface 50 of the vehicle 2. More specifically, in the digital model 20, the surface 50 is represented by the virtual surface 50’. The virtual surface 50’ is a representation in the digital model 20 of the surface 50 of the vehicle 2 in Fig. 1. The virtual surface 50’ typically defines the base substrate layer in the digital model 20. Hence, throughout the description, the virtual surface 50’ is different from the surface 50, which denotes the physical surface of the vehicle 2.

[0056] The digital model 20 is e.g. created by a digital model creation unit (not shown) configured to generate the digital model. The controller 90c is here configured to use a CAD software to generate the digital model. The digital model 20 of the surface 50 and the vehicle 2 may also be obtained by using 3D scanning, which is a commonly used technology. The 3D scanning is then performed by the 3D scanner. To this end, the digital model 20 serves as the nominal reference for subsequent operations and is based on a CAD model of the surface 50 and the vehicle 2. As illustrated in Fig. 2, the method 100 also comprises a step 130 of positioning, in the digital model 20, a digital image 22a on the virtual surface 50'. The digital image represents an image 22b to be painted on the surface 50.

[0057] Optionally, the method 100 may comprise a step 120 of providing the digital image 22a to be painted as the image 22b on the surface 50. The digital image 22a is here an integral part of the digital model 20. The digital image 22a is aligned with the nominal surface profile of the virtual surface 50’ in the digital model 20. As such, the digital model 20 comprises at least one digital image 22a with image details. The digital image 22a is intended to be visualized (painted) on the surface 50 as the image 22b. The digital image 22a is defined by image details in the form of an array of pixels. The image details here refer to specific visual elements or design attributes of the digital image 22a to be printed. For example, the image details define a decal. In one example, the image details provide a print design designed to produce a 3D visual effect on the surface 50. Merely as an example, the digital image 22a and the image 22b are here illustrated in the form of a tree.

[0058] The positioning of the digital image 22a on the virtual surface 50’ in the digital model 20 ensures that the digital image 22a is correctly aligned relative to the nominal surface profile of the surface as represented by the virtual surface 50’ in the digital model 20. The digital image 22a is typically embedded on the virtual surface 50’ in the digital model 20. More specifically, the digital image 22a is mapped onto the virtual surface 50’ in the digital model 20. The image details and the digital image 22a can be considered as other layers in the digital model 20.

[0059] Typically, the digital model 20 comprises geometry data 24 of the virtual surface 50’ and data of the embedded digital image 22a relative to the virtual surface 50’. The data can contain e.g. positioning data, such as spatial positioning data, and / or data about the different layers of the digital model

[0060] 20. It should be noted that the digital model 20 supplied to the processing circuitry 92 can have the digital image 22a and image details embedded in the CAD geometry. In other examples, the digital image 22a is converted into a CAD geometry and subsequently wrapped onto the virtual surface 50’. As such, the step of providing no the digital model 20 containing data of the surface geometry of the virtual surface 50’ of the vehicle 2 can be performed in several different manners. In one example, a customer of the vehicle 2 provides the digital image 22a intended to be painted and visualized on the surface 50 in a digital file. In such example, the digital model 20 is updated based on the digital image 22a intended to be painted and visualized on the surface 50, in which the processing circuitry 92a further embeds the image details on the virtual surface 50’ in the digital model 20. Such operation is performed to provide an example of the digital model 20 containing data of the surface geometry of the virtual surface 50’. Such configuration of the digital model 20 also comprises the digital image 22a intended to be visualized on the surface 50. Such configuration of the digital model 20 may also contain positioning data indicative of the digital image 22a embedded on the virtual surface 50’ in the digital model 20. The digital model 20 may also be configured in a ready-to-use state and stored in the memory 94. These operations are typically performed by the processing circuitry 92. Alternatively, or in addition, the processing circuitry 92 obtains a ready-to- use digital model 20. That is, the digital model 20 contains a user-prepared digital model 20 comprising data of the surface geometry of the virtual surface 50’, the digital image 22a with the image details, and positioning data indicative of the digital image 22a embedded on the virtual surface 50’ in the digital model 20. The provided digital model 20, the digital image 22a with the image details are typically stored in the memory 94 of the control system 90, such as in the controller 90a.

[0061] In summary, steps no and 130, or steps no to 130, of the method 100 first create the digital image 22a in the form of a nominal image based on a nominal CAD model in the form of the digital model 20, as is illustrated in Fig. 2 in conjunction with Fig. 3. Subsequently, as illustrated in Fig. 2 in conjunction with Fig. 3, the method 100 comprises a step 140 of determining, in the digital model 20, one or more nominal reference points 24a to 24n on the virtual surface 50’, i.e. on the surface as represented in the digital model 20. For ease of reference, Fig. 3 illustrates the step 140 along an intended painting path 23 of the nozzle arrangement 14. The intended painting path 23 is typically determined based on the details of the digital image 22, the surface geometry of the virtual surface 50’ from the digital model 20 in combination with data about the configuration of the robot system 1 and the dimensions and configurations of the painting head 11, etc. Note that step 140 refers to an operation in relation to the digital model 20 and the virtual surface 50’, as illustrated.

[0062] The nominal reference points 24a to 24n correspond to specific zones 25a to 25n that potentially affect the visual representation of the digital image 22 a. That is, the zones may typically have an impact on the visual representation of the painted image on the surface 50, as compared with the digital image 22a on the virtual surface 50’. More specifically, the nominal reference points 24a to 24n correspond to zones 25a to 25n on the surface 50 that potentially affect the visual representation of the digital image 22a on the surface 50. The zones 25a to 25n on the surface 50 correspond to locations with distinct geometric features. Examples of such zones 25a to 25n on the surface 50 can be locations with distinct geometric features, such as edges, ridges, or holes. For example, in Fig. 3, zone 25a indicates an end edge of the roof surface, zone 25b indicates a first edge of a hole 26, zone 25c indicates a second edge of the hole 26, and zone 23d indicates an opposite edge of the roof surface. These types of zones 25a to 25n include any locations that impact the visual representation of the digital image 22a when painted as the image 22b on the surface 50. As such, the nominal reference point 24a locates zone 25a, the nominal reference point 24b locates zone 25b, the nominal reference point 24c locates zone 25c, and the nominal reference point 24b locates zone 25b.

[0063] In another example, the zones 25a to 25n on the surface 50 include the edges of the roof, defining the nominal reference points 24a and 24b, and e.g. the sunroof borders, defining the nominal reference points 24b and 24c. In other examples, the zones can indicate edges of the hood, door boundaries, and mounting holes for logos or emblems, and the like. Including the edges as zones affecting the visual representation of the digital image 22a maybe useful, for example, if the actual hood is longer than the CAD model. The digital image 22a will typically cover an area from one end to another on the virtual surface 50’ in the digital model 20, but not when the image 22b is printed on the surface 50. However, by using reference points for the end and start of the hood, the digital image 22a can be modified to fit the surface 50. If the digital image 22a is not adjusted, an unpainted area, the same size as the deviation, at the end of the hood will be visible.

[0064] For the above reasons, the control system 90, such as the controller 90a, is configured to determine one or more nominal reference points 24a to 24n associated with the positioned digital image 22a on the surface in the digital model 20, i.e. the virtual surface 50’. The nominal reference points 24a to 24n indicate zones 25a to 25n on the surface potentially affecting the visual representation of the digital image 22a. The term “associated with” here means that the reference points are identified in connection to the digital image 22a after it has been positioned on the virtual surface 50’ in the digital model 20.

[0065] Accordingly, in order to improve the visual result of the painted image on the surface 50, the method 100 is configured to take any zones on the actual surface 50 into consideration, and then manipulate the digital image 22a defining the underlying image based on the actual location of these zones on the surface 50. Typically, the actual printing surface of the surface 50 is a distinct geometric shape that is easily detected by the 3D scanner, but also visually important.

[0066] As such, as illustrated in Figs. 2 and 3, the method 100 also comprises a step 150 of determining an actual painting surface geometry 27 of the vehicle from measured geometry data of the surface 50. Note that step 150 in Fig. 3 refers to an operation of the spatial data acquisition device 18 in relation to the surface 50 of the vehicle 2. Fig. 4 schematically illustrates an example of determining an actual painting surface geometry of the surface 50 using the spatial data acquisition device 18. The actual painting surface geometry 27 defines the actual surface profile of the surface 50. Byway of example, step 150 involves using the spatial data acquisition device 18, such as a 3D scanner, to obtain precise measurements of the actual surface profile of the surface 50. The 3D scanner is moved over the surface 50, e.g. as illustrated in Fig. 4, in which the 3D scanner measures surface geometry and also detects the zones on the surface 50. More specifically, the 3D scanner is operated to detect the actual location of the zones on the actual surface 50 of the vehicle 2. In this manner, method 100 uses the 3D scanner to measure the surface 50 and the vehicle 2, and then identify the zones from the measured geometry data. The measured geometry data is transferred to the processing circuitry 92. Hereby, the control system 90, such as the controller 90a knows the actual location of each reference point of each zone. As the measured data contains data indicative of the actual surface 50, the measured data can also be considered as actual surface data. The measured data contains spatial data of the actual surface 50. The 3D scanner can be controlled to scan the surface 50 and the zones based on data from the digital model 20 and / or from other suitable commands from the control system 90, such as from the controller 90a.

[0067] More specifically, the 3D scanner identifies the locations of one or more actual reference points of the zones on the surface 50. In order to distinguish from the reference points of the zones on the virtual surface 50’, the actual reference points are referenced with numerals 24a! to 24m, as illustrated in Figs. 3 and 4, while the zones on the surface 50 are referenced with numerals zones 25a! to 25m. As such, the nominal reference points 24a to 24n represent locations on the virtual surface 50’ in the digital model 20, while actual reference points 24a! to 24m correspond to the measured positions on the surface 50.

[0068] Accordingly, based on the determined actual painting surface geometry 27, the method 100 comprises a step 160 of determining the locations of one or more actual reference points 24a! to 24m of the zones 25a! to 25m on the surface 50. The actual reference points 24a! to 24m serve as references for the locations of the corresponding zones 25a! to 25m on the surface 50 as the nominal reference points 24a to 24n for the zones on the virtual surface 50’, but reflect the true, measured positions based on the measured surface geometry of the surface 50. Accordingly, a measured actual reference point is different from the digital model nominal reference point. The actual reference points are detected e.g. by the 3D scanner. In Fig. 3, this is depicted for illustrative purposes by that the hole 26 has slightly moved to the left (see step 160), and the scan using the 3D scanner has detected a change in location of the reference points between the virtual surface 50’ and the surface 50 (see step 160 of Fig. 3). It should be noted that the method typically compares all actual reference points 24a! to 24m of the zones 25a! to 25m with the nominal reference points 24a to 24n of the zones 25a to 25m

[0069] Subsequently, the method 100 comprises a step 170 of determining a deviation 29 between the determined one or more nominal reference points 24a to 24n and the determined locations of the one or more actual reference points 24a! to 24m. The calculation of the deviation 29 is here also further performed using a best-fit algorithm, which is applied and configured to optimize the alignment of the digital image 22a to the surface 50 by minimizing the differences in the locations between corresponding nominal reference points 24a to 24n and actual reference points 24a! to 24m. Accordingly, the method 100 determines the deviation 29 between the determined one or more nominal reference points 24a to 24n in the digital model 20 and the determined locations of the one or more actual reference points 24a! to 24m in the actual surface 50, as e.g. illustrated in Fig. 3.

[0070] As such, by comparing the actual reference points 24a! to 24m and nominal reference points 24a to 24n, and applying a best fit algorithm, the deviation between the “actual” image 22b and the “nominal” digital image 22a, as represented in the digital model 20, can be determined based on the deviation 29 between the actual and nominal reference points. Moreover, the method 100 comprises a step 180 of adapting the digital image 22 in response to the determined deviation 29. Adapting the digital image 22 in response to the determined deviation 29 typically involves aligning the position of the digital image 22 on the virtual surface 50’ in the digital model 20 relative to the determined actual painting surface geometry 27 of the surface 50. In this example, the adaptation process may involve adding or removing one or more pixels in the digital image 22 to ensure precise alignment with the actual surface geometry defining the surface profile of the surface 50. To this end, the control system 90 and / or the controller 90a is configured to adapt the digital image 22a in response to the determined deviation 29 by aligning the position of the digital image 22a in the digital model 20 relative to the determined actual painting surface geometry 27 of the surface 50.

[0071] Finally, the method comprises a step 190 of controlling the robot system 1 to paint on the surface 50 according to the adapted digital image 22 c. For example, the control system 90, such as the controller 90a within the robot system 1 executes the adapted painting instructions containing the adapted digital image 22c, ensuring that the paint is accurately applied in accordance with the adapted digital image 22c.

[0072] As such, the method 100 comprises using the adapted digital image 22c when painting the image 22b on the surface 50. The adapted digital image 22c is thus an integral part of a painting sequence instruction to the robot control system, as described above in relation to Fig. 1. The painting sequence instruction for the nozzle arrangement 14 for painting the image 22b and the image details on the surface 50 may also comprise determining the paths and / or movement of the robot arm arrangement 12 in a conventional manner.

[0073] As mentioned above, the step 150 of determining an actual painting surface geometry 27 is performed by the spatial data acquisition device 18 in the form of a 3D scanner. However, the spatial data acquisition device can be any type of scanner, such as a 3D scanner or a 2D line scanner. Hence, while a more advanced 3D scanner can be used in some examples to gather all the measured data for determining the actual painting surface 27 in one scanned image, the method may in other examples combine the use of a 2D line scanner with the movement of the robot arm arrangement 12. Combining the position of the robot arm arrangement 12 with the gathered surface geometry data from the 2D line scanner provides for acquiring 3D data of the surface 50.

[0074] Moreover, as mentioned above, the step 170 of determining a deviation 29 between the determined one or more nominal reference points 24a to 24n and the determined locations of the one or more actual reference points 24a! to 24m is here performed using a best fit algorithm. However, the method for determining a deviation between the one or more nominal reference points 24a to 24n and the one or more actual reference points 24a! to 24m may include the use of various algorithms tailored to the specific type of geometry involved. For geometries where extensive measured data is available across all dimensions, a standard best-fit algorithm may be applied, which calculates the optimal alignment by minimizing the overall differences between the nominal and actual reference points. For surfaces that are flat or feature distinct edges, a global best-fit approach may not always yield suitable accurate alignment results. In such cases, the method may involve utilizing an edge detection algorithm that identifies the first data point corresponding to the edge of the surface 50. Similarly, for geometric features such as ridges, the method may include applying a peak detection algorithm, which identifies the peak data point representing the ridge. Such specialized algorithms are selected based on the geometric characteristics of the surface 50 to ensure accurate alignment of the digital image 22a with the actual surface geometry of the surface 50. For example, when aligning a digital image 22a to a car roof with a pronounced ridge running down the center, the method may utilize a peak detection algorithm to identify the highest point of the ridge on the actual surface profile. The absolute reference point associated with the peak is then compared to the corresponding nominal reference point in the digital model 20. Any deviation between the nominal peak and the actual peak is calculated, and the digital image 22a is adjusted accordingly to an adjusted digital image, ensuring that the image 22b aligns correctly with the ridge when painted onto the car roof. For flat sections of the roof, an edge detection algorithm maybe used to locate the first data point on the edge, ensuring that the digital image 22a aligns with the edge features of the roof.

[0075] The step 180 of adapting the digital image 22a in response to the determined deviation comprises aligning the position of the digital image 22a on the surface 50’ in the digital model 20 relative to the determined actual painting surface geometry 27 of the surface 50. Hereby, the method 100 aligns the position of the digital image 22a relative to the surface 50 based on the actual surface geometry 27, rather than to the virtual surface 50’, thereby reducing the likelihood of misalignment and enhancing the overall quality of the painted image 22c.

[0076] Moreover, aligning the position of the digital image 22a on the virtual surface 50’ in the digital model 20 relative to the determined actual painting surface geometry 27 here comprises adding or removing one or more pixels in the digital image 22a. Adding or removing pixels allows the digital image 22a to be finely tuned to match the actual surface profile defined by the measured surface geometry in the form of the actual painting surface geometry 27. For instance, if the actual painting surface geometry 27 of a car’s roof is slightly longer than the nominal digital model suggests, the method 100 may involve extending the digital image 22a by adding rows of pixels. To maintain the visual integrity of the image, the last row of pixels in the affected area is copied and added as a new row, effectively stretching the image to fit the extended surface. Conversely, if the actual painting surface geometry 27 is shorter than expected, rows of pixels maybe removed from the digital image 22, typically by omitting the last row in the affected area, thereby shortening the image to match the reduced surface length. Such pixel-level adjustment ensures that the digital image accurately conforms to the actual dimensions of the surface 50, thereby reducing visual distortion or misalignment during the painting process. In addition, or alternatively, the rows of pixels might also be added or removed in the middle of the affected area, depending on the application of the method and system.

[0077] In addition, or alternatively, adapting the digital image 22a in response to the determined deviation 29 comprises aligning the digital image 22a on the virtual surface 50’ in the digital model 20 in discrete segments. Moreover, each segment here corresponds to an actual reference point 24a! on the surface 50. Then, the method 100 comprises adjusting the dimensions of the digital image 22a within each segment to compensate for deviations in the positions of the reference points relative to the actual painting surface geometry. Such segmentation can better accommodate complex surface geometries, such that each segment of the image 22b accurately conforms to the corresponding part of the surface 50. Such segment-based adaptation approach may also be useful when multiple features need to be aligned. In one example, the method 100 involves painting a stripe across the full roof of the vehicle, where the roof includes distinct geometric features such as an antenna hole and a sunroof opening. The digital image 22a intended to represent the stripe is aligned using reference points set at the start of the roof, at the antenna, at the start and end of the sunroof, and at the end of the roof. If a deviation is detected where the antenna hole is closer to the sunroof than initially expected, the method 100 compensates for this deviation by expanding the portion of the digital image 22a applied before the antenna and shrinking the portion of the digital image 22a between the antenna and the sunroof. Such segment-based adaptation ensures that the digital image is accurately aligned with all features across the surface, even when multiple reference points are involved.

[0078] It should be noted that although the examples above are described in relation to a surface 50 in the form of a roof surface of a vehicle 2, the surface may be another type of surface of the vehicle, including e.g. a hood surface, a side panel surface and the like. The top surfaces of a vehicle, including the roof and hood, are particularly suitable for painting because these surfaces are large, and relatively flat areas. Other types of surfaces maybe side panels, rear end and trunk lid. It should also be noted that the surface 50 may be a surface of any type of a 3D object. Other examples of vehicles besides passenger cars may be airplanes, underwater vessels and the like. In such examples, the surface 50 can be either the exterior surface or the interior of the vehicle.

[0079] The steps of the method 100 are typically performed by the control system 90 comprising one or more controllers 90a, 90b and the processing circuitry 92. As such, the processing circuit 92 is configured to perform the steps no to 190.

[0080] The above presentation of the robot system 1 should also be regarded as disclosing a control system 90 having one or more controllers 90a, 90b, 90c for controlling the robot system 1, for instance using the controller 90a and the processing circuitry 92. In addition, there is disclosed a controller 90a comprising processing circuitry 92 configured to provide the digital model 20 containing data of the surface geometry of the surface; provide the digital image 22a to be painted as the image 22b on the surface; position, in the digital model 20, the digital image 22a on the virtual surface 50’; determine, in the digital model 20, one or more nominal reference points 24a to 24n associated with the positioned digital image 22a on the virtual surface 50’, the nominal reference points indicating zones 25a to 25n on the surface 50 potentially affecting the visual representation of the digital image 22a on the surface 50; determine the actual painting surface geometry 27 of the surface 50 from measured geometry data of the surface 50; based on the determined actual painting surface geometry, determine locations of one or more actual reference points 24a! to 24m of the zones 25a! to 25m on the surface 50; determine the deviation 29 between the determined one or more nominal reference points and the determined locations of the one or more actual reference points; adapt the digital image 22a in response to the determined deviation between the determined one or more nominal reference points and the determined locations of the one or more actual reference points; and control the robot system 1 to paint on the surface 50 according to the adapted digital image. Thanks to the present invention, as described herein in relation to the figures, there is provided an improved preparation of the painting and execution of the painting of images on surfaces of complex 3D objects. The invention is at least partly based on the insight that where e.g. inkjet is used to print an image onto an object, the image is generated offline, based on a nominal CAD model. However, the actual surface of the object may not be identical to the nominal object reflected in the CAD model. Hence, the CAD model may have deviations, which might cause a mismatch between the image and the object it is printed on. E.g. if a car roof is slightly longer than the CAD, the image / paint will end before the actual car body ends, leading to poor quality, unnecessary reprogramming and image regenerating. The methods, systems and controllers of the present invention provides for managing these challenges by modifying the image based on a comparison of one or more measured actual reference points 24a! to 24m with one or more corresponding nominal reference points 24a to 24n on the digital model 20.

[0081] By identifying and measuring the position of key reference points (also referred to as anchor points) in the digital model 20, e.g. corresponding to a CAD model, and on the surface profile of the object, the image can be modified to fit the actual measured geometry of the surface, and without changing the digital model and / or completely regenerating the image. Moreover, by using a scanner, as mentioned herein, key geometric points can be identified in the actual part, including e.g. the start and end of the part, or a hole like a sunroof or a logo. After the object has been scanned and the deviation between the nominal points and actual measured points are measured, the image can be modified by e.g. stretching the image to fit the actual part.

[0082] As described herein, the disclosure also relates to the control system 90, comprising controllers 90a, 90b, configured to execute the method 100 according to the above examples. The disclosure also relates a computer program comprising instructions to cause the controller 90 to execute the method of any of the above examples. It should be noted that the controller, as described herein, may include a microprocessor, microcontroller, programmable digital signal processor or another programmable processor device. The processing circuitry may also include a microprocessor, microcontroller, programmable digital signal processor or another programmable processor device, or instead, include an application specific integrated circuit, a programmable gate array or programmable array logic, a programmable logic device, or a digital signal processor. Where the processing circuitry includes a programmable device such as the microprocessor, microcontroller or programmable digital signal processor mentioned above, the processor may further include computer executable code that controls operation of the programmable device.

[0083] Even though the invention has been described with reference to specific exemplifying embodiments thereof, many different alterations, modifications and the like will become apparent for those skilled in the art. Also, it should be noted that parts of the system and method may be omitted, interchanged or arranged in various ways, the system and method yet being able to perform the functionality of the present invention.

[0084] Additionally, variations to the disclosed embodiments can be understood and effected by the skilled person in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

Claims1. A computer-implemented method (ioo) for controlling painting of a surface (50) of a three-dimensional (3D) object (2) using a robot system (1), the robot system comprising a nozzle arrangement (14) with a set of controllable nozzles (16) for depositing paint on the surface, wherein the method comprises:- providing (no) a digital model (20) containing data of a surface geometry of a virtual surface (50’), the virtual surface representing a virtual counterpart of the surface;- positioning (130), in the digital model, a digital image (22a) on the virtual surface (50'), the digital image representing an image (22b) to be painted on the surface;- determining (140), in the digital model, one or more nominal reference points (24a to 24n) associated with the positioned digital image on the virtual surface, the nominal reference points indicating zones (25a to 25n) potentially affecting the visual representation of the digital image;- determining (150) an actual painting surface geometry (27) of the surface from measured geometry data of the surface;- based on the determined actual painting surface geometry, determining (160) locations of one or more actual reference points (24a! to 24m) of the zones (25a! to 25m) on the surface;- determining (170) a deviation (29) between the determined one or more nominal reference points and the determined locations of the one or more actual reference points;- adapting (180) the digital image in response to the determined deviation; and- controlling (190) the robot system to paint on the surface according to the adapted digital image.

2. The method according to claim 1, wherein determining the actual painting surface geometry of the surface is performed by a spatial data acquisition device.

3. The method according to any one of the preceding claims, wherein determining the deviation between the determined one or more nominal reference points and the determined locations of the one or more actual reference points is performed using a best-fit algorithm.

4. The method according to any one of the preceding claims, wherein adapting (180) the digital image in response to the determined deviation comprises aligning the position of the digital image in the digital model relative to the determined actual painting surface geometry of the surface.

5. The method according to claim 4, wherein aligning the position of the digital image in the digital model relative to the determined actual painting surface geometry of the surface comprises adding or removing one or more pixels in the digital image.

6. The method according to claim 4 or claim 5, wherein adapting the digital image in response to the determined deviation comprises aligning the digital image on the surface in discrete segments, each segment corresponding to one or more actual reference points on the surface, and adjusting the dimensions of the digital image within each segment to compensate for deviations in the positions of the reference points relative to the actual painting surface geometry.

7. The method according to any one of the preceding claims, wherein the zones (25a to 25n, 25a! to 25m) on the surface correspond to locations with distinct geometric features.

8. A controller (90a) comprising processing circuitry (92) configured to execute the method of any one of claims 1 to 7.

9. A robot system (1), comprising: a nozzle arrangement (14) configured to deposit paint onto a surface (50); a robot arm arrangement (12) configured to move the nozzle arrangement over the surface; and one or more controllers (90, 90a, 90b) according to claim 8.

10. The robot system according to claim 9, wherein the controller is configured to determine the actual painting surface geometry of the surface using a spatial data acquisition device.

11. The robot system according to claim 9 or claim 10, wherein the controller is configured to determine a deviation between the one or more nominal reference points (24a to 24n) and the one or more actual reference points (24a! to 24m) using a best-fit algorithm.

12. The robot system according to any one of claims 9 to 11, wherein the controller is configured to adapt the digital image in response to the determined deviation by aligning the position of the digital image in the digital model relative to the determined actual painting surface geometry of the surface.

13. The robot system according to claim 12, wherein the controller is configured to align the position of the digital image in the digital model by adding or removing one or more pixels in the digital image.

14. A computer program product comprising program code for performing, when executed by a controller, the method of any of claims 1 to 7.

15. A non-transitory computer-readable storage medium comprising instructions, which when executed by a controller, cause the controller to perform the method of any of claims 1 to 7.

Images