Method for providing pixel values ​​using a region of interest, image sensor, and 3D imaging system based on optical triangulation.

Abstract

A method relating to providing pixel values ​​of image sensor pixels of a second ROI based on a first ROI, and an imaging system for 3D imaging based on optical triangulation including an image sensor. [Solution] A first ROI is used by the image sensor to provide a first pixel value of the image sensor pixels of the first ROI resulting from a first exposure. A second ROI is determined according to a predetermined relationship that the second ROI has with respect to the first ROI (702). A second exposure (703) causes at least the image sensor pixels of the second ROI to acquire a second pixel value. The second ROI is used to provide a second pixel value of the image sensor pixels of the second ROI for further processing by the image sensor (704).

Description

Technical Field

[0001] Embodiments of this specification relate to methods and arrangements for an image sensor for providing pixel values using a region of interest (ROI) in particular, and a 3D imaging system based on optical triangulation using the image sensor.

Background Art

[0002] Industrial vision cameras and systems for factory and logistics automation may be based on 3D machine vision where three-dimensional (3D) images of objects, etc. are captured. By a 3D image, it is meant an image that includes information such as "height" or "depth", not just information regarding intensities and / or colors of only two-dimensional (2D) pixels as in a normal image, or at least not only such information.

[0003] Generally, each pixel in an image captured by a camera has a position in image sensor coordinates corresponding to the position of what the camera and image sensor are capturing in the real world, or more specifically, information about the light from the real-world position sensed by the image sensing elements of the image sensor and which image sensing element corresponds to the pixel. Typically, what is sensed is reflected light from what is being captured, for example, from an object. Depending on the camera and system, what kind of light is used, and how the illumination is provided, the sensed light may contain various information about the position from which the light was reflected, such as the position on the object being captured. Thus, each pixel in the captured image has a position in image sensor coordinates corresponding to a real-world position, such as the position on the object. The sensed light may also contain additional information about the position and the properties of the object at that position, such as information from the intensity, color, reflectivity, scattering, etc. Many 3D machine vision cameras or systems, or 3D imaging systems in general, for 3D imaging are based on the fact that multiple 2D images are sequentially captured by the camera's image sensor, generally continuously, during the scanning of an object. Each such 2D image may contain 3D information about the object's 2D profile, and therefore the entirety of such 2D images may contain 3D information about the entire object, from which a 3D image of the entire object may be formed. The 3D image may be represented by a "point cloud" where each point corresponds to a position on the object and is associated with a 3D coordinate relative to that point. Each point may also be associated with further information about the point, such as color or other features associated with the point on the corresponding object.

[0004] When a pixel has a 3D position, not just a 2D position, that pixel is sometimes called a voxel.

[0005] Line scan image data is generally obtained when an object is scanned using a light plane onto which lines of light are projected, and the reflected light from the object is measured, resulting in image data being scanned or provided line by line at a time. A special case of 3D imaging by scanning is 3D imaging based on optical triangulation, where structured light, e.g., a specific pattern, typically a light plane or "sheet of light," is used, and the object is scanned through and / or by this light plane. Lines of light are projected onto the object during scanning, corresponding to where the sheet or plane intersects the object. Lasers are often preferred, but other light sources that can provide structured light, such as a light plane, may also be used, e.g., light sources that can provide light that remains focused and does not spread out too much, such as light-emitting diodes (LEDs). Instead of a light plane corresponding to a "sheet of light," a light plane corresponding to the edge of illumination, i.e., a light edge, may be used.

[0006] 3D machine vision systems are often based on optical triangulation. In such systems, there is a light source that illuminates an object with a specific pattern of light, typically structured light such as the optical plane described above. This type of 3D machine vision system or device is sometimes called a system or device for 3D imaging based on optical or optical plane triangulation, or simply laser triangulation when laser light is used. A line of light projected onto an object is imaged by a camera, i.e., the light reflected from the object is imaged. Along the line of light, 3D features are captured by optical triangulation, corresponding to a profile of the object with height information. By scanning the entire object in this way, corresponding to line scanning and including the movement of the line and / or the object, it is possible to capture 3D features of the entire object, corresponding to multiple 2D profiles of the object, on which a 3D image of the object can be formed as discussed above. To generate a profile image of the object during scanning, reflected light from the object is captured by the camera's image sensor, and in particular, peaks in the intensity of the reflected light are detected in the image data. The peak occurs at a position corresponding to the location on the object where the incident light corresponding to the light line is reflected from the object. The position of the detected peak in the image is mapped to the location on the object where the light that produced the peak was reflected, according to the optical triangulation configured and set up by the system.

[0007] Peak detection algorithms typically operate in the digital domain, and their goal is to find the central location of the peak light distribution at the subpixel resolution. This means that an image with intensity peaks containing analog pixel values ​​corresponding to light sensed by sensor elements corresponding to pixels in the image sensor must first be read from the image sensor and then converted analog-to-digital (A / D). Typically, pixel values ​​in the same row are read simultaneously from the image sensor, i.e., then A / D converted in parallel and then stored and / or processed digitally. When pixels in a row are read, another readout, for example, of pixels in the next row, is performed, and so on. That is, there are parallel readouts of pixels in the same row, and such pixel rows are read sequentially, i.e., sequentially, and A / D converted.

[0008] Generally, it is desirable to eliminate or reduce latency, thereby facilitating or enabling higher throughput from 3D imaging systems, for example, how quickly a 3D imaging system based on optical triangulation can provide 3D images of an object, and / or supporting or better supporting high-speed or faster applications.

[0009] The intensity peaks corresponding to the object's profile typically form a pattern of lines or a segment of lines in the image captured by the image sensor pixels. Only a small fraction of the image sensor pixels have information about these intensity peaks. If all image sensor pixels are read out, there are therefore many pixels and readout operations performed that do not contribute to providing useful information, i.e., do not have information about the intensity peaks that could be used to determine the location of the intensity peaks at subpixel resolution.

[0010] Avoiding reading pixel values ​​from image sensor pixels that do not contain useful information can therefore save time and enable faster readout of the image sensor and the imaging system using the image sensor, and thereby faster operating speeds.

[0011] Existing solutions to this problem are based on using a region of interest (ROI), covering the peak of intensity while excluding other image sensor pixels, and then reading only the pixels of such an ROI. A simple ROI can be formed by excluding image sensor rows that do not contain any pixels with pixel values ​​exceeding a certain threshold, leaving the rest to form the ROI.

[0012] Another, more sophisticated and efficient solution is to use a so-called Window Around Maximum (WAM), or Window At Maximum (WAM), as an ROI. The WAM is formed around the intensity peak in each image sensor column, such that, for example, in each column, the WAM covers the pixel with the maximum pixel value and several of the closest pixels on either side. Thus, it is possible to consider the WAM as a "smaller" WAM for each column, and these together form the overall WAM, which therefore follows an intensity line corresponding to the profile of the object captured by the image. The maximum pixel value in a column can be found on the image sensor without performing a complete readout from the image sensor, or by performing a rough, fast readout first, only to determine the pixel with the maximum pixel value in each column.

[0013] EP4266673A1 discloses a method and arrangement for providing pixel values ​​for reading from pixels of an image sensor based on WAM. The WAM is formed by the image sensor such that a WAM of a specific width is located at or around the pixel in each image sensor row where the analog pixel value during exposure first reaches or exceeds a threshold pixel value in the row. [Prior art documents] [Patent Documents]

[0014] [Patent Document 1] EP4266673A1 [Patent Document 2] EP4220075A1 [Overview of the project] [Problems that the invention aims to solve]

[0015] In view of the foregoing, an object of the present invention is to provide one or more improvements or alternatives to the prior art, in particular, to reduce delays involved in the operation of a 3D imaging system and / or image sensor based on or for 3D imaging based on laser triangulation. [Means for solving the problem]

[0016] According to a first aspect of the embodiments of this specification, the objective is achieved by a method for providing a pixel value of an image sensor pixel of a second ROI based on a first region of interest ROI, performed by an image sensor. The first ROI is used by the image sensor to provide a first pixel value of an image sensor pixel of the first ROI. The first pixel value is obtained as a result of a first exposure of at least the image sensor pixel of the first ROI. Each of the ROIs partially covers the entirety of an image sensor pixel, thereby partially covering the image sensing area of ​​the image sensor including the entirety of the image sensor pixel. The image sensor determines the second ROI according to a predetermined relationship that the second ROI has with respect to the first ROI. The image sensor exposes at least the image sensor pixels of the second ROI to a second exposure, thereby causing the image sensor pixels of at least the second ROI to acquire a second pixel value. The image sensor uses the second ROI to provide a second pixel value of an image sensor pixel of the second ROI for further processing by the image sensor.

[0017] According to a second aspect of the embodiments of this specification, the object is an image sensor for providing a pixel value of an image sensor pixel of a second ROI based on a first ROI, wherein the first ROI is achieved by the image sensor, which is used by the image sensor to provide a first pixel value of an image sensor pixel of the first ROI. The first pixel value is obtained as a result of a first exposure of at least the image sensor pixel of the first ROI. Each of the ROIs partially covers the entirety of an image sensor pixel, thereby partially covering the image sensing area of ​​the image sensor including the entirety of the image sensor pixel. The image sensor is configured to determine the second ROI according to a predetermined relationship that the second ROI has with respect to the first ROI. The image sensor is further configured to expose at least the image sensor pixels of the second ROI to a second exposure, thereby causing the image sensor pixels of at least the second ROI to acquire a second pixel value. Furthermore, the image sensor is configured to use the second ROI to provide a second pixel value of an image sensor pixel of the second ROI for further processing by the image sensor.

[0018] According to a third aspect of the embodiments of this specification, the objective is achieved by a method performed by a three-dimensional 3D imaging system based on optical triangulation. The method relates to the reading of data from the imaging system. The imaging system is for 3D imaging of an object and includes a camera having an image sensor according to a second aspect, and one or more light sources for providing one or more illumination of the object during imaging. The 3D imaging system performs a first readout of first data from the image sensor. The first data is based on the first pixel value of an image sensor pixel of a first ROI. The first pixel value is generated as a result of the operation of the imaging system including the image sensor, such that the first exposure of the image sensor was under first illumination of the object provided by the one or more light sources. Then, the 3D imaging system performs a second readout of second data from the image sensor. The second data is based on the second pixel value of an image sensor pixel of a second ROI. The second pixel value is generated as a result of the operation of the imaging system including the image sensor, such that the second exposure of the image sensor was under second illumination of the object provided by the one or more light sources.

[0019] According to a fourth aspect of the embodiments of this specification, the objective is achieved by an imaging system for 3D imaging of an object based on optical triangulation. The imaging system includes a camera having an image sensor according to a second aspect, and one or more light sources for providing one or more illuminations of the object during imaging. The imaging system is configured to perform a first readout of first data from the image sensor. The first data is based on the first pixel value of an image sensor pixel of a first ROI. The first pixel value is generated as a result of the operation of the imaging system including the image sensor, such that the first exposure of the image sensor was under first illumination of the object provided by the one or more light sources. The 3D imaging system is further configured to perform a second readout of second data from the image sensor. The second data is based on the second pixel value of an image sensor pixel of a second ROI. The second pixel value is generated as a result of the operation of the imaging system including the image sensor, such that the second exposure of the image sensor was under second illumination of the object provided by the one or more light sources.

[0020] According to a fifth aspect of the embodiments of this specification, the objective is achieved by one or more computer programs that, when executed by one or more processors, cause an image sensor according to the second aspect and / or an imaging system according to the fourth aspect to perform the method according to the first aspect and / or the method according to the third aspect.

[0021] According to a sixth aspect of the embodiments of this specification, the objective is achieved by one or more carriers comprising one or more computer programs according to the fifth aspect.

[0022] When 3D image data and associated intensity peaks are present within the image, solutions such as those described in the background, using ROIs, e.g., WAM, can be applied. However, it has proven that this is not always the case with recent advances in optical triangulation-based 3D imaging systems, for example, when 2D data is to be associated with captured 3D data. Such solutions are disclosed, for example, in EP4220075A1. The 2D data may be read from images that do not contain 3D image data, and therefore there are no intensity peaks on which ROIs, such as WAM as described in the background, can be determined. Nevertheless, it is not the 2D data from every pixel of the image that we are interested in using and therefore associating with the 3D image data. The situation appears to be further complicated by the fact that optical triangulation-based 3D imaging systems are typically based on scanning of continuously moving objects or cameras, resulting in a sequence of image frames at a specific scanning rate, i.e., there is still movement of position involved when data is captured from different image frames.

[0023] The embodiments of the present specification and their aspects disclosed above provide solutions thereto. ROIs such as the first ROI used for reading 3D image data from a 3D image frame, e.g., a WAM, can be "reused" for 2D image data in a 2D image frame associated with the 3D image data by determining a second ROI according to a predetermined relationship that the second ROI has with the first ROI. This enables a fast and efficient determination of the second ROI, particularly in relation to the use for reading 2D image data that is or will be associated with the 3D image data. For example, due to the occurrence of movement of an object being imaged between image frames, it may happen that some adjustment or minor change of the first ROI attracts attention, e.g., to make the second ROI more appropriate and / or more efficient. Such a change may be realized by a suitable predetermined relationship including a specific offset that the second ROI has with respect to the first ROI and / or a reduced or increased spread. Determining and using the second ROI as in the embodiments of the present specification enables a significant saving in the reading time compared to reading 2D image data of the entire 2D image frame. Also, if the second ROI has a reduced spread compared to the first ROI, this enables a saving in the reading time even compared to the reading of 3D image data using the first ROI while still allowing appropriate 2D data associated with the 3D data to be continuously captured.

[0024] Examples of embodiments of the present specification will be described in more detail with reference to the accompanying schematic drawings briefly described below.

Brief Description of the Drawings

[0025] [Figure 1] FIG. schematically shows an example of a prior art 3D imaging system based on optical triangulation. [Figure 2] FIG. schematically shows a simplified example of a prior art imaging system based on optical triangulation. [Figure 3A] This diagram schematically shows an example of an image sensor with an image sensing area. [Figure 3B] This is another schematic diagram of the image sensing area. [Figure 3C] This diagram schematically shows the distribution of light as an example of the distribution of light around the intensity peak. [Figure 3D] This diagram schematically illustrates two different types of ROIs related to the image sensing area. [Figure 4A] This diagram provides a schematic and illustrative illustration of a second ROI, referred to as ROI2 and ROI2', in relation to the first ROI, referred to as ROI1 and ROI1'. [Figure 4B] This diagram provides a schematic and illustrative illustration of a second ROI, referred to as ROI2 and ROI2', in relation to the first ROI, referred to as ROI1 and ROI1'. [Figure 4C] This diagram provides a schematic and illustrative illustration of a second ROI, referred to as ROI2 and ROI2', in relation to the first ROI, referred to as ROI1 and ROI1'. [Figure 4D] This diagram provides a schematic and illustrative illustration of a second ROI, referred to as ROI2 and ROI2', in relation to the first ROI, referred to as ROI1 and ROI1'. [Figure 5] This figure provides a schematic and illustrative example of the second ROI, ROI2. [Figure 6] This figure schematically illustrates an example of a 3D imaging system based on optical triangulation, in which embodiments of this specification may be performed and / or configured to perform embodiments of this specification. [Figure 7] This flowchart schematically illustrates and shows the first method according to the embodiments of this specification. [Figure 8] This is a schematic block diagram illustrating an embodiment of how an image sensor may be configured to perform the first method and action described in relation to Figure 7. [Figure 9]This flowchart schematically illustrates and shows a second method according to the embodiments of this specification. [Figure 10] This schematic block diagram illustrates embodiments of how parts of one or more devices of an imaging system may be configured to control the 3D imaging system 605 to perform the second method and / or action described in relation to Figure 9, and / or to cause the 3D imaging system 605 to perform the second method and / or action described in relation to Figure 9. [Figure 11] This is a schematic diagram showing embodiments relating to a computer program and its carrier. [Modes for carrying out the invention]

[0026] The embodiments described herein are exemplary. It should be noted that these embodiments are not necessarily mutually exclusive. Components from one embodiment may be implicitly assumed to be present in another embodiment, and how those components may be used in other exemplary embodiments will be apparent to those skilled in the art.

[0027] Figure 1 schematically shows an example of an imaging system of the type described in the background art, namely, an imaging system 105 for 3D machine vision or simply 3D imaging based on optical triangulation, i.e., imaging for capturing information about the 3D features of an object using a camera 130. System 105 is shown in the figure in normal operation, i.e., typically after calibration has been performed and therefore after the system has been calibrated. System 105 is configured here to perform optical triangulation in the form of triangulation of a sheet of light, as described in the background art. System 105 further includes a light source 110, for example, a laser, for illuminating the object to be imaged with a specific pattern of light corresponding to the sheet of light, typically structured light such as the optical plane 111 shown in the figure. The light is typically laser light, but alternatively, it may be light from, for example, one or more light-emitting diodes (LEDs). An alternative to, but with a similar effect, an optical edge, i.e., the edge of the illuminated area. The generated light and illumination are typically provided through one or more lenses of the camera 130, for example, to focus the light. Furthermore, the camera 130 is typically configured and positioned to have a focal plane aligned with the light plane 111, in other words, aligned with the light plane 111, based on the so-called Scheimpflug principle or Scheimpflug focusing. In this way, object reflections occurring in the light plane are focused on the image sensor. In the example shown, the objects to be imaged are exemplified by a first object 120 in the form of an automobile and a second object 121 in the form of a gear structure. When the light plane 111 is incident on an object, this corresponds to the projection of the light plane 111 onto the object, which may be seen when the light plane 111 intersects with the object. For example, in the example shown, the light plane 111 produces a line of light 112 on the first object 120. Light is reflected by an object, more specifically by a portion of the object at the intersection, i.e., at the line of light 112 in the shown example. Camera 130 includes an image sensor (not shown).The camera and image sensor are positioned relative to the light plane 111 such that the light plane 111, when reflected by an object, becomes incident light on the image sensor through the camera 130. The image sensor is typically implemented as a chip to sense the incident light and convert it into image data as intensity values ​​resulting from the light sensed by, for example, sensing elements corresponding to pixels in the image sensing area of ​​the image sensor. For example, in the shown example, the light plane 111 is reflected toward the camera 130 and image sensor in a line of light 112 on a portion of the roof of the car object 120, and the camera 130 and image sensor thereby generate and provide image data containing information about the portion of the car roof. Using a setup including the geometric arrangement of the system 105, for example, knowledge of how the image sensor coordinates relate to real-world coordinates such as x, y, z in a Cartesian coordinate system, which is related to the object being imaged and its context, the image data may be converted into information about the 3D features of the object being imaged, such as its 3D shape or profile, in a preferred format. The 3D features, for example, information regarding the 3D shape or profile, may include data describing the 3D features in any preferred format.

[0028] By moving the object to be imaged, such as a light source 110 and / or a first object 120 or a second object 121, so that multiple parts of the object are illuminated and reflected light is produced on the image sensor, for example, by scanning the object, image data describing the more complete 3D shape of the object may be generated, corresponding to multiple consecutive profiles of the object, such as shown profile images 141-1 to 141-N of the first object 120, each profile image showing the contour of the first object 120 as reflected by the light plane 111 when the image sensor of the camera 130 senses the light that gives rise to the profile image. As shown in the figure, the light source 110 and camera 130 are usually stationary, and a conveyor belt 122 or the like may be used to move the object across the light plane 111. Alternatively, the light plane and / or camera 130 may be moved over the object so that all parts of the object, or at least all parts facing the light source 110, are illuminated, and the camera can receive light reflected from different parts of the object that are desirable to image.

[0029] As can be understood from the above, for example, an image frame of a first object 120, provided by the camera 130 and its image sensor, may result in one of the profile images 141-1 to 141-N. As described in the background art, each position of the contour of the first object shown in one of the profile images 141-1 to 141-N is typically determined based on the identification of intensity peaks in the image data captured by the image sensor and finding the locations of these intensity peaks by, for example, one or more peak-finding algorithms. System 105 and typical peak-finding algorithms are typically configured to search for intensity peaks pixel by pixel in each image frame. As shown in the figure, if the sensor coordinates are u, v, then u may be used to indicate a position in such a row that lies along the image sensor row, for example, corresponding to an image sensor column. Correspondingly, v may be used to indicate a position in such a column that lies along the image sensor column, for example, corresponding to an image sensor row.

[0030] For each position u in the image frame, for example, that position u may be searched for along v by a peak discovery algorithm as described above, and the identified peaks in the image frame may result in one of the profile images 141-1 to 141-N as shown in the figure. The profile image is formed by image points in coordinate system 143, u, v, t based on sensors related to real-world coordinates such as x, y, z in coordinate system 123, as described above. The entire image frame and profile image may be used to create a 3D image of the first object 120 in the form of such a "point cloud" as described in the background art, where each point corresponds to a position on the object and is associated with 3D coordinates relative to that point.

[0031] As a development toward embodiments of this specification, the circumstances described in the background art will first be explained in more detail.

[0032] Imaging systems based on optical triangulation, like all technologies, continue to evolve and are being extended, for example, to the capture of 2D data in relation to the captured 3D data, the latter including the discovery of intensity peaks, as discussed above and in the background technology. Such solutions are disclosed, for example, in EP4220075A1.

[0033] Figure 2 schematically shows a simplified example of a prior art imaging system that may be configured to operate as in EP4220075A1, i.e., to acquire 2D image data and associate it with 3D image data based on optical triangulation. One difference compared to a conventional 3D "only" imaging system based on optical triangulation is the presence of one or more additional light sources to provide separate illumination for 2D imaging and thus light captured in the 2D image data associated with the 3D image data. In the figure, such additional light sources are exemplified by a first light source 210 that provides a first light 211 for 3D imaging based on optical triangulation, in addition to a further second light source 240 that provides a second light 241. A solution like EP4220075A1 may be implemented by software controlling a conventional 3D imaging system based on optical triangulation, such as system 105, which has the added second light source 211. Therefore, except for the second light source 240, the imaging system 205 may structurally correspond to the imaging system 105, but it is shown here in a more simplified form, where one or more additional light sources, such as the second light source 240, are used to generate 2D image data associated with 3D image data.

[0034] Therefore, in addition to the second light source 240, the imaging system 205 includes features corresponding to those discussed above in relation to Figure 1, such as a camera 230 having an image sensor 231. The figure also shows an object 220 to be imaged within the field of view (232) of the camera 230, the object being illuminated by, for example, a first light 211 in the form of a light plane 211 and a second light 241. There may be cases where both the first and second lights illuminate the object 220 simultaneously, or where only one of the lights illuminates the object 220 simultaneously, for example, when a 3D image frame is captured by the camera 221, the first light 211 may illuminate the object 220, and when an image frame with only 2D image data is captured by the camera 231, the second light 241 may illuminate the object 220. The figure also shows an example of a computing device 233, such as a computer, which may be connected to the camera 231 in such a system, for example, to receive and process image data from the camera 231, and / or to control the camera and / or the light source 210, 240, etc.

[0035] When 3D image data and associated intensity peaks are present in the image, solutions such as those described in the background art, using ROIs, e.g., WAM, may be applied; however, this has proven not to apply to all embodiments disclosed in EP4220075A1. In some embodiments thereof, 2D data is read from an image that does not contain 3D image data, for example, because the first light for 3D imaging, typically a laser, is temporarily turned off and only light for 2D imaging is present. With respect to such images, there are therefore no intensity peaks on which ROIs, such as WAM as described in the background art, can be determined. Nevertheless, it is not the 2D data from all pixels of the image that we are interested in using and therefore relating to the 3D image data. Therefore, it is desirable to be able to read only this 2D image data and not read 2D image data from all pixels.

[0036] The situation can be further complicated by the fact that, at least at first glance, there is some movement of the object between image frames.

[0037] The 2D data of interest is typically the 2D data associated with the peaks in intensity from image frames with 3D image data that were captured most temporally before and / or after the image frame with the 2D image data.

[0038] To visualize the situation where separate 2D data exists in 2D image frames between 3D image frames, the profile images 141-1 to 141-N shown in Figure 1 can be considered, as they each correspond to 3D image frames. In this case, separate 2D image frames with 2D image data lie between 3D image frames, and therefore, correspondingly, between the profile images 141-1 to 141-N shown in Figure 1, but the separate 2D image frames do not have "lines" formed by intensity peaks corresponding to the object's profile.

[0039] Embodiments of this specification are at least in part based on the recognition that the change in the position of the image sensor between a 3D image frame and a 2D image frame having 2D image data associated with the 3D image data is, in practice, usually so small that it can be ignored, i.e., does not need to be compensated for the inclusion of different image frames. Alternatively, the change in the position of the image sensor due to movement between an image frame with a 3D image and an image frame with its associated 2D image data may be calculated and / or estimated from the available information of the 3D imaging system being used. This may be done when it is known how the system is or will be set up to be operational for scanning and capturing the image frames. In that case, the change in the position of the image sensor may be determined, for example, by calculating and / or estimating from the geometric relationship between the camera, its field of view, and the light used for 3D imaging, typically the location of the light plane, the frame rate at which the camera operates, and the speed of movement of objects.

[0040] Therefore, the position of the image sensor for the 2D image data relative to the 3D image data is near the location of the intensity peak of the 3D image data, and / or within a certain offset from these locations.

[0041] As a result, an ROI, e.g., WAM, used for reading 3D image data from a 3D image frame can be "reused" at least to some extent for reading 2D image data related to the 3D image data, even if the 2D image data belongs to a different 2D image frame. If a change in the ROI is of interest, e.g., due to movement between image frames or for other reasons, this may be compensated for by small changes such as, for example, an offset of the ROI along a pixel column, and / or typically a height, i.e., an expansion or contraction of the ROI coverage along the column. A contraction of ROI coverage may be of interest, for example, when only a single pixel value per column is used as 2D image data for the 3D position in the same column. In that case, the ROI may be contracted to the height of a single pixel, thereby providing pixel values ​​from only a single pixel per column for reading the 2D image data. This allows for significant savings in read time compared to reading 2D image data from the entire image frame, and therefore from all rows, and also significant savings compared to reading 3D image data when the full, uncontracted ROI is used.

[0042] In summary, "reusing" the ROI of a 3D image frame for one or more 2D image frames has several advantages. As explained above, the ROI for a 3D image frame is still relevant, even though it is for another image frame, especially when the 2D image frames are captured in close temporal proximity between them. In the case of some 2D image frames, they are typically captured consecutively immediately after a 3D image frame that has the 3D image data to which the 2D data of the 2D image frame is associated. The ROI of the 3D image frame is already available and used by the image sensor. Modifications to the ROI that may attract interest concern offsetting and / or stretching and / or shrinking, which are simple modifications that can be easily performed directly by the image sensor on the image sensor based on the available ROI for the 3D image data. The 2D image data associated with the 3D image data can be read out without having to read out all the 2D image data from the 2D image frame. As a result, the 2D image data can be provided as fast as, or even faster than, the 3D image data.

[0043] Embodiments of this specification relate to providing a second pixel value of an image sensor pixel of a first ROI, for example, a second ROI, such as a second WAM based on a first WAM, including the 2D image data. The first ROI is an ROI used by the image sensor to provide a first pixel value of an image sensor pixel of the first ROI, for example, a first pixel value including the 3D image data to which the 2D image data is associated. The first pixel value is generated as a result of a first exposure of at least the image sensor pixel of the first ROI, but typically of the entire image sensor, and corresponds to, for example, the pixel value of the 3D image frame as described above.

[0044] The second ROI is determined according to a predetermined relationship that the second ROI has with respect to the first ROI. When the second ROI is determined based on the first ROI, then the relationship that the second ROI has with respect to the first ROI, for example, having a specific offset and / or expansion or reduction of coverage, is already determined, i.e., predetermined. Before or after the second ROI is determined, the image sensor pixels of the image sensor, or at least the image sensor pixels of the second ROI, are exposed to a second exposure, thereby causing the image sensor pixels of at least the second ROI to acquire a second pixel value, for example, the 2D image data. The second pixel value of the image sensor pixels of the second ROI may then be provided by using the second ROI. The second pixel value is provided for further processing by the image sensor, for example, to provide the second pixel value for readout from the image sensor. Further processing may alternatively include, for example, the image sensor processing the pixel value and then outputting the resulting information for readout from the image sensor.

[0045] To enable a deeper understanding of WAM as ROI, the circumstances discussed above, and the embodiments herein, more detailed examples of WAM as ROI are now illustrated and discussed in the context of images and image sensors.

[0046] Figure 3A schematically shows an example of an image sensor 331 having an image sensing area 351 formed by the entirety of pixels, shown as squares. The pixels of the image sensor 331 correspond to light sensing elements and may be of a conventional type. In the example, the entirety of pixels is arranged in N rows and M columns, where N = 12 in the shown example. Thus, there are 352-1...352-M columns and 354-1...352-12 rows. The shown figure is divided along the rows.

[0047] In the image sensing area 351, an intensity peak line 361 is drawn, and the darker pixels are formed by 5 pixels per column, thus indicating an ROI with a height 362 of 5 pixels, which is a WAM centered on the intensity peak line 361. Of course, such a line or WAM cannot be seen by visual inspection of the corresponding real image sensing area 351. The intensity peak line 361 and WAM are drawn solely to facilitate understanding how the intensity peaks and WAM relate to each other and to the image sensor. The intensity peak line 361 corresponds to a location where the intensity peak may be located at subpixel resolution and may occur as a result of exposure of the image sensor 331 when the image sensor 331 is part of a camera in a 3D imaging system based on optical triangulation, such as one of the imaging systems 105, 205. If an image based on the entire pixels is read from the image sensor 331 after the image sensor 331 has been exposed to a line of light intensity having an intensity peak according to the intensity peak line 361, the line of light intensity may become visible in the image, and the intensity peak line 361 may be detected therefrom. Thus, such lines of intensity and the intensity peak line 361 may be caused by reflected light from an object when the object is illuminated by a line of light during optical triangulation as described above in relation to Figure 1.

[0048] In practice, each pixel can typically hold only a single pixel value corresponding to an intensity value. The WAM for each column 352 is centered around the detected pixel that holds the maximum pixel value for each column, and these pixels are marked with thick lines in the figure. The sum of the column-by-column WAMs for all columns M forms the overall WAM centered around the intensity peak line 361.

[0049] In this example, there are N = 12 pixel rows 0...11 corresponding to image sensor rows 354-1...354-12, and M pixel columns 1...M corresponding to image sensor columns 352-1...352-M.

[0050] Typically, pixel values ​​are read in parallel from the image sensor 331, row by row; that is, the pixel values ​​of pixels in the same row are read in parallel and then converted from analog to digital (A / D). As shown in the figure, M columns 352 may be connected to M parallel A / D converters (ADCs) 356. An alternative technique may instead be to read pixels with the same WAM position in parallel; that is, all the dark pixels in Figure 3A, i.e., all the pixels corresponding to the WAM, may be read by five consecutive parallel readouts. A detailed example of such an implementation is disclosed, for example, in EP4266673A1, mentioned in the background art.

[0051] After A / D conversion, and therefore in the digital domain, there may be some processing and / or calculation circuits, such as the calculation circuit 357 schematically shown in the figure, that operate on the resulting digital image data, for example, pixels relating to values ​​from each column and / or from multiple columns. Furthermore, there may be additional memory elements included for temporarily storing pixel values ​​and intermediate data during such processing and / or calculations. Such memory elements may enable the image sensor to perform processing and calculations on pixel values ​​from all pixels of the WAM.

[0052] Furthermore, the image sensor 331 includes, for example, some input / output (I / O) circuit 358 that is involved in the input for controlling the image sensor, enables the reading of image data from the image sensor 331 after A / D conversion, and / or enables the reading of information obtained as a result of the processing performed by the calculation circuit 357, for example. When the image sensor provides something for reading from the image sensor, this is via an I / O circuit such as the I / O circuit 358.

[0053] The figure also shows the control circuit 359 involved in controlling the image sensor 331 and parts thereof, for example, for addressing and / or selecting columns and / or rows and / or pixels to be A / D converted. The control circuit 359 may also be involved in temporarily storing information such as ROIs such as WAM in use, and which pixels are included in WAM.

[0054] Figure 3B is another schematic diagram of the image sensing area 351. Image sensor coordinates u and v are shown instead of individual pixels as in Figure 3A. In the figure, u corresponds to the column coordinate and v corresponds to the row coordinate. The ROI in the form of a WAM, shown by dark pixels in Figure 3A, is shown here as a WAM 363, corresponding to the area between the WAM lower edge line and the WAM upper edge lines 363a and b. The intensity peak line 361 is also shown for reference. Instead of 12 rows as in Figure 3A, the image sensing area 351 is drawn here in a wider N rows. The ROI height 362 is also shown in this figure, but in a broader form without specifying that it is 5 pixels as in Figure 3A. Figure 3B still shows that the ROI height 362 is the same for all columns and all column coordinates u, but the ROI, when it is a WAM, follows and covers the intensity peak line 361, so the position of the ROI, in this case the position of the WAM, is different along u.

[0055] Figure 3C schematically shows the distribution of light 364 as an example of the distribution of light around the intensity peak. The distribution is shown along axis v, i.e., along columns, for example, column m depicted in Figure 3B.

[0056] The intensity values ​​of pixels along column m are shown as small black squares in the figure. As can be seen, the intensity values ​​represent intensity I, belong to different rows, i.e., are at different row coordinates v, and therefore vary along the column. The intensity values ​​occur according to the distribution of light, a normal distribution that is typically Gaussian "bell" shaped.

[0057] In practice, when the ROI is WAM, the ROI height 362 is chosen so that, with respect to the imaging system being used, the result provides sufficient coverage of the light distribution 364, and therefore the peak discovery algorithm can discover the actual center position of the light distribution 364 at subpixel resolution, i.e., it can utilize the light distribution 364 to discover the peak position along v at subpixel resolution. The peak discovery algorithm is generally used outside the image sensor after the 2D image data has been read from the image sensor. As shown in the figure, as shown in Figure 3A, the actual peak position has the maximum intensity value and has some offset from the center position of the pixel around which the WAM is centered. In the example shown, the ROI height 362 is 5 pixels, but this is just an example and is usually a little lower. In practice, when an ROI that is WAM is used, a height that covers at least about 75% of the energy of the intensity peak should usually be used. With respect to a typical sensor and setup, this may mean an ROI height in the range of 7 to 25 pixels. However, there may be situations in which ROI heights less than 7 and greater than 25 may also be used.

[0058] Figure 3D schematically shows two different types of ROIs related to the image sensing area 351. ROI 363, also denoted here as ROI1, has already been discussed above and is a WAM. ROI 363', also denoted as ROI1', is also shown, which is a more "ordinary" type of rectangular ROI. Each of the ROIs, ROI1 and ROI1', partially covers the entire image sensor pixel, or in other words, partially covers the image sensing area 351. Both cover the intensity peak line 361 shown for reference with some margin, so that there is sufficient coverage of the light distribution around the peak, for example, as discussed above. Both ROI1 and ROI1' are examples of first ROIs that may be used in embodiments herein and are used as examples below. ROI1, being a WAM, may be considered more efficient than ROI1' in covering what is relevant because it tracks the intensity peak line 361.

[0059] Here again, we refer to embodiments herein in which the second ROI is determined according to a predetermined relationship that the second ROI has with respect to the first ROI.

[0060] Figures 4A to 4D schematically illustrate and illustrate different second ROIs, called ROI2 and ROI2', determined according to different illustrative predetermined relationships, in relation to a first ROI called ROI1 and ROI1'. ROI1 and ROI1' are drawn for reference in the examples shown and correspond to ROI1 and ROI1' discussed above in relation to Figure 3D, namely, ROI1 is the first ROI 363 and ROI1' is the first ROI 363'. These are merely two different examples of the first ROI, and it should be noted that the embodiments herein may, of course, be used with other types of first ROIs, as well as / or first ROIs whose shapes are determined and / or positioned differently from those in the examples herein.

[0061] ROI1 and ROI1' may have been determined as discussed above, i.e., to cover the distribution of light around the intensity peaks resulting from 3D imaging based on optical triangulation. More broadly, a first ROI such as ROI1 or ROI1' may have been used by an image sensor such as image sensor 331 to provide a first pixel value of the image sensor pixel for the first ROI resulting from a first exposure of the image sensor pixel.

[0062] In the examples shown, the second ROI, ROI2, and ROI2' are drawn with thick dashed lines, while the first ROI, ROI1, and ROI1' are drawn with finer dotted lines. Each ROI is located between the lines shown, as should be understood. The type of predetermined relationship that the second ROI has with the first ROI is indicated above each example.

[0063] Figure 4A shows an example where a given relationship is that ROI2 is a copy of ROI1 and thus covers the same image sensor pixels as ROI1. Thus, Figure 4A shows, on the left, the second ROI, ROI2 463a, which is a copy of the first ROI, ROI1 363, and on the right, the second ROI, ROI2' 463a', which is a copy of the first ROI, ROI1' 363'.

[0064] Figure 4B shows an example where a given relationship is such that ROI2 can be shaped like ROI1 but is positioned with a specific offset relative to ROI1, thereby covering at least some image sensor pixels different from those covered by ROI1. In the example, the specific offset is in the direction along the image sensor row and is downward, but carefully, it may be in a different direction. Thus, Figure 4B shows on the left a second ROI, ROI2 463b, which is the first ROI, ROI1 363, with a specific offset, and on the right a second ROI, ROI2 463b', which is the first ROI, ROI1' 363', with a specific offset.

[0065] Figure 4C shows an example where a given relationship can be shaped such that ROI2 is similar to ROI1, but with a specific increased spread along that particular direction, covering more pixels along that particular direction than ROI1. Thus, Figure 4C shows, on the left, a second ROI, ROI2 463c', which is a first ROI, ROI1 363', with a specific increased spread, and on the right, a second ROI, ROI2 463c', which is a first ROI, ROI1' 363', with a specific increased spread.

[0066] A specific increased spread of an ROI might include, for example, one or more pixels being covered in the direction of increase, such that one or more additional pixels along a column are covered by a second ROI.

[0067] Figure 4D shows an example where a given relationship is such that ROI2 can be shaped like ROI1, but with a specific reduced spread along that particular direction, such that ROI2 covers fewer pixels along that particular direction than ROI1. Thus, Figure 4D shows on the left a second ROI, ROI2 463d, which is a first ROI, ROI1 363, with a specific reduced spread, and on the right a second ROI, ROI2 463d', which is a first ROI, ROI1' 363', with a specific reduced spread.

[0068] A specific decrease in the spread of an ROI might include, for example, one or more pixels being covered in the direction of decrease, such that one or more pixels along a column are covered by a second ROI.

[0069] In the examples in Figures 4C to 4D, the specific increased spread and the specific decreased spread are shown in both directions along the image sensor row, but alternatively, they may be in only one direction and / or different directions, such as a spread in only "upward" or "downward" along the row.

[0070] Furthermore, increased or decreased spreads, as shown in the examples in Figures 4C and 4D, can of course be combined with offsets, as shown in the example in Figure 4B. An example of this follows.

[0071] Figure 5 schematically illustrates and illustrates a second ROI 565 or ROI2, which is an example of a second ROI or ROI2 having a predetermined relationship that is both a reduced spread and an offset relative to the first ROI, also called ROI1. As should be understood, Figure 5 is a slightly more detailed example of a second ROI related to a first ROI than the more simplified examples in Figures 4A–4D. As can be seen, the first ROI 363, ROI1 is the same ROI illustrated and discussed above in relation to Figure 3A. In Figure 5, the same pixels are darkened to represent ROI1, as in Figure 3A. Furthermore, below ROI1 are the pixels of ROI2 as the second ROI 565, indicated by a darker color and labeled reference number 565.

[0072] The figure shows that ROI2 565 has the same shape as ROI1 but is narrower and therefore has a reduced spread along the column. Instead of ROI1 having a height of 362 pixels with a height of 5 pixels, ROI2 565 has a height of 1 pixel. As seen in the figure, ROI2 565 also has a 3-pixel offset "downward" in the figure.

[0073] In practice, using ROI1, which is WAM as shown and may be available after it has been used to capture 3D image data of a 3D image frame, ROI2 565 can be provided simply by considering the center pixel line of ROI1 as WAM and offsetting it by 3 pixels, which thus corresponds to a predetermined relationship that ROI2 has with respect to ROI1 in this case. The image sensor 331 may be configured to do this for 2D image frames following the 3D image frame in which ROI1 is provided and used. Thus, it relates to changing the existing ROI, ROI1, to ROI2 565 according to a predetermined relationship before applying ROI2 for the next single or multiple exposures.

[0074] Figure 6 schematically shows an example of a 3D imaging system 605 based on optical triangulation in which embodiments of the foregoing may be performed and / or configured to perform embodiments of the foregoing. The figure also shows the context in which an image sensor 631 configured to work according to embodiments of the foregoing may be used. Imaging system 605 corresponds to imaging system 205 but may be further configured to perform embodiments of the foregoing. For example, in the figure, image sensor 631 is part of camera 630, which is included in imaging system 605 and used for 2D and 3D imaging in exactly the same way as imaging system 205 and may perform embodiments of the foregoing in that context. Image sensor 631 may correspond to image sensor 331 configured to perform embodiments of the foregoing and may be an image sensor implemented as considered in EP4266673A1 in order to facilitate the use of WAM as ROI.

[0075] Therefore, the diagram shows the following:

[0076] Object 620 may correspond to object 120 or 220, and is shown to be at least partially located within the field of view 632 of camera 630.

[0077] Typically, the first light source 610 of the imaging system 605 is configured to illuminate the object 620 with a first light 611 for 3D imaging based on optical triangulation, in the form of structured light, such as a line of light, like a laser line, that is produced on the object 620, reflected by the object 620, and captured by the camera 630 having an image sensor 631. Another example of structured light that can be used as the first light is an optical edge, i.e., the edge of an area or part of the illumination. For example, light from an LED rather than a laser is possible. The illumination is in the vertical direction, i.e., substantially parallel to the z-axis, in this example. However, one or more other directions of illumination are of course available in the embodiments herein.

[0078] A further second light source 640 for the imaging system 605 to provide a second light 641 which may be used to generate 2D image data associated with 3D image data from 3D imaging, as disclosed in EP4220075A1.

[0079] A computing device 633, such as a computer, may be part of the imaging system. The computing device 633 may be connected to the camera 630 and / or light sources 610, 640 to receive and process image-related data and information from the camera 631, and / or to control the imaging system 605 and / or the camera 631 and / or its image sensor 631, and / or light sources 610, 640. The computing device 633 may be configured to perform and / or participate in some of the actions relating to some embodiments of this specification, as will be further described below. Alternatively or additionally, the computing device 633 or its corresponding function may be fully or partially integrated into the same unit as the camera 630, in which case the camera 630 may be considered to have this function.

[0080] Similar to conventional 3D imaging by optical triangulation, the object 620 may be illuminated by a first light 611, and the image may be captured by a camera 630 having an image sensor 631. Furthermore, the object 620 may be imaged under illumination of a second light 641, and the image may be captured by a camera 630 having an image sensor 631 to generate 2D image data associated with 3D image data as disclosed in EP4220075A1.

[0081] Accordingly, similar to conventional optical triangulation-based imaging systems, the imaging system 605 may be configured to move the first light source 610 together with a second light source 640, and / or move the object 620 relative to the illumination by the light sources 610, 640, for example, so that different consecutive parts of the object 620 are illuminated by the first light 611 and the second light 641 at different consecutive moments. The object may be moved as part of the transport of the object, for example, by a conveyor belt. After reflection from the object 620, the light is sensed by the image sensor 631. Each image frame is associated with each moment when the image frame was sensed, i.e., captured, and each part of the object 620 from which the image sensor 631 sensed the reflected light 611 at each moment.

[0082] Image frames provided by camera 630 and / or image sensor 631, and / or information derived from image frames, such as the location of intensity peaks and associated 2D image data, may be transferred to an external device, such as computing device 633, for further processing.

[0083] Figure 7 is a flowchart illustrating a schematic first method according to embodiments of this specification, which has already been shown above but is disclosed in some more detail here. The following actions, which may form a method, are for providing pixel values ​​of image sensor pixels of a second ROI based on a first ROI. The following method and / or actions may be performed by an image sensor, such as image sensor 331 or 631, or by an imaging system, such as imaging system 605, and / or other preferred devices. Hereinafter, it is assumed that the image sensor exemplified by image sensor 331 performs the method. The first ROI, which may also be called ROI1, may be, for example, ROI1 365 or 365'. Hereinafter, ROI1 365 may be used as a non-limiting example of the first ROI. The second ROI, which may also be called ROI2, may be, for example, one of ROI2 465a~d, 465a~d', or 565. For simplicity, below, the second ROI may be referred to as ROI2 465, and ROI2 465 may therefore be used as a non-restrictive example of the second ROI, referring to any one of ROI2 465a-d.

[0084] As can be understood from the above discussion, the image sensors of the embodiments herein should be suitable for use as image sensors in cameras of imaging systems for 3D imaging of objects based on optical triangulation. Furthermore, as should be understood, an image sensor pixel corresponds to a photosensitive element that senses light during exposure. Each pixel value acquired by each image sensor pixel during exposure typically corresponds to the cumulative amount of light received by each image sensor pixel during exposure. The amount and pixel value typically correspond to a measurement of the intensity of the light incident on the image sensor pixel, or more specifically, its photosensitive portion, during exposure.

[0085] Therefore, the first ROI is the ROI used by the image sensor performing the method, preferably such as image sensor 331, to provide the first pixel value of the image sensor pixels of ROI1 365 that results from the first exposure of at least the image sensor pixels of ROI1 365, but typically all image sensor pixels, i.e., the entire exposure of the image sensor pixels.

[0086] Each of the ROIs, namely the first ROI and the second ROI, partially covers the entire image sensor pixels, thereby partially covering an image sensing area such as the image sensing area 351 of the image sensor 331, which includes the entire image sensor pixels. Therefore, each ROI may cover only a subset of all pixels of the image sensor and correspond to an area or contiguous group of pixels in the image sensing area 351.

[0087] Please note that the following actions may be performed in any preferred order, and / or may overlap completely or partially in time, where preferable.

[0088] Action 701 The image sensor 331 may acquire information identifying a first ROI, exemplified here by ROI1 365. Typically, this information may be acquired internally by the image sensor 331 and automatically obtained from the aforementioned use of the first ROI, for this means that the image sensor 331 actually has available information about the first ROI from use, and thereby has information to identify it, or in other words, the first ROI is already known to the image sensor 331. If another device is performing the method, the information identifying the first ROI may be acquired, for example, by receiving this information from the image sensor 331 using the first ROI. As already shown above, ROI1 365 may be an ROI such as WAM, which is conventionally used for reading 3D image data from an image frame, thereby avoiding reading image data from pixels that do not contain useful information.

[0089] Action 702 The image sensor 331 determines ROI2 465 according to a predetermined relationship that ROI2 465 has with respect to ROI1 365.

[0090] The aforementioned predetermined relationship itself may be set and / or selected by the user, and / or set by specific settings of the image sensor and / or by operating modes that the image sensor is configured or controlled to operate in accordance with.

[0091] Examples of the aforementioned relationship between ROI1 and ROI2, and how ROI2 may relate to ROI1, were discussed above in relation to Figures 4 and 5. For example,

[0092] A given relationship may be that ROI2 465b can be shaped like ROI1 365, but is positioned on the image sensing area 351 with a specific offset, such as offset 467 relative to the first ROI1 365, thereby covering at least some image sensor pixels that are different from those covered by ROI1 365.

[0093] The specific offset may be in any direction on the image sensing area 351, but typically corresponds to a column-direction offset. The specific offset may be predetermined and / or, for example, when the image sensor 331 is used by an imaging system based on optical scanning of an object, for example system 605, it may correspond to or be based on the distance that the object being imaged, for example object 620, moves or moves between the first exposure and the second exposure.

[0094] As used herein, "shape-determined like the first ROI" includes not only the same shape and size, but also the same shape but different size, such as having a reduced or increased spread compared to the first ROI, for example, having height. For example, covering the same, fewer, or more pixels in each column as the first ROI.

[0095] Furthermore, a predetermined relationship may be such that ROI2 465, such as ROI2 465c or 465d, is shaped on the image sensing area 351 with a specific reduced or increased spread along a particular direction, similar to ROI1 365, but with ROI2 465 covering more or fewer pixels along that particular direction than ROI1 365.

[0096] In general, the specific direction may, in principle, be any direction on the image sensing area 351. A reduced or increased spread of ROI1 along the specific direction means that the number of pixels in ROI1 along that direction is reduced or increased, and usually corresponds to a reduction or increase in the height or width of ROI1. Typically, the reduction or increase is of the height of ROI1, or in other words, ROI2 covers more or fewer pixels in each pixel row than ROI1.

[0097] In a special case, the given relationship is that ROI2 465a, for example, is a copy of ROI1 365, and thereby covers the same image sensor pixels as ROI1 365.

[0098] Action 703 The image sensor 331 exposes at least the image sensor pixels of ROI2 465 to a second exposure, thereby causing at least the image sensor pixels of ROI2 465 to acquire a second pixel value.

[0099] Note that the first and second exposures are not necessarily exposures of the same pixels, but in practice, the first exposure is usually of all image sensor pixels, and the second exposure is another, generally subsequent, exposure of all image sensor pixels. Typically, there was a readout of pixel values ​​from the ROI1 365 image sensor pixels between exposures.

[0100] Action 704 The image sensor 331 uses ROI2 465 to provide a second pixel value of the image sensor pixel of ROI2 465 for further processing by the image sensor 331.

[0101] The further processing described above may include the image sensor 331 providing a second pixel value for readout from the image sensor, and / or the image sensor performing a calculation involving the second pixel value and providing the result for readout from the image sensor.

[0102] As already shown above, the first ROI, i.e., ROI1 such as ROI1 365, may cover the intensity peaks on the image sensing area 351 corresponding to the intensity peaks of the 3D image frame when the image sensor 331 is used in a 3D imaging system based on optical triangulation, such as the imaging system 605. Therefore, ROI1, such as ROI1 365, may partially cover at least several rows of image sensor pixels in the image sensing area 351 and the same number of pixels in each such row. The same number of pixels may correspond to the spread or height of ROI1 or WAM. The same number of pixels may be predetermined or determined by the image sensor and / or user by selection from a predetermined different number of pixels based on some input to and / or settings for the image sensor 331. ROI1, such as ROI 365, may be determined by the image sensor 331 to cover the image sensor pixels based on a first pixel value of the image sensor pixels. ROI1, such as ROI 365, may be determined by the image sensor 331 to cover each intensity peak resulting from the first exposure along each pixel line, for example, a column, of the image sensing area 351. Each intensity peak may result from the first exposure of an entire image sensor pixel, such as the entire image sensing area 351 of the image sensor 331. As a result of the first exposure, the image sensor pixels of the image sensing area 351 acquired the first pixel value, for example, during the first illumination state of the image sensing area 351. Thus, each intensity peak may be discovered by looking at the first pixel value resulting from the first exposure. The image sensor 331 may have discovered a pixel with the locally largest first pixel value by, for example, searching the image sensor pixels along each pixel column, along the pixel line in a particular direction. At that time, the image sensor 331 may form a first ROI by covering each pixel having such a locally maximum first pixel value and the nearest neighbor pixels on both sides of each such pixel, typically along the particular direction, for example, one or more nearest neighbor pixels on each side of the pixel having the locally maximum first pixel value. ROI1, such as ROI 365, may therefore be determined by the image sensor 331 to cover the same number of pixels in each pixel line around the respective intensity peaks of each pixel line. This should result in ROI1 covering the same number of pixels along each pixel line in the particular direction, for example, the same number of pixels in each pixel row, and covering the respective intensity peaks in the row. In the case of intensity peaks belonging to sensed reflected light lines from 3D imaging based on optical triangulation, ROI1 is understood to cover the reflected light lines when the reflected light lines were captured by the image sensor 331 during the first exposure. In these embodiments, the first ROI may therefore be a whole WAM that may be formed from a window WAM around several local maximums, one local WAM each in each pixel line, typically in each pixel row, or may correspond to a whole WAM. Therefore, the local WAM of each pixel line may cover the pixel in that pixel line that has the locally largest first pixel value, and the nearest pixel in that pixel line, and the same number of pixels in each pixel line are covered by the first ROI.

[0103] After action 704, a second ROI, ROI2, i.e., the same ROI2, may be used for one or more further exposures, corresponding to further execution of actions 703-704 for a third exposure, etc.

[0104] Furthermore, please note that in the second execution of the method, the first ROI, ROI1, may be the second ROI, ROI2, from the previous execution of the method.

[0105] In the case of multiple executions, such as several consecutive executions of the method, the given relationship may remain the same. With respect to embodiments that use the given relationship that results in a modified ROI, the same given relationship means, for example, that the second ROI in a second execution of the method is different from the second ROI in a previous execution of the method.

[0106] Figure 8 is a schematic block diagram illustrating an embodiment of how the image sensor 800 may be configured to perform the first method and action described in relation to Figure 7. The image sensor 800, or image sensing device, or more broadly, simply one or more devices, may correspond to devices already mentioned above for performing embodiments herein, such as for performing the method and / or action described in relation to Figure 7. The image sensor 800 may correspond to either the image sensor 331 or 631. What is shown in Figure 8 may be considered a different diagram of how the image sensor 331 may be configured. Thus,

[0107] The image sensor 800 may include processing means, such as one or more hardware modules including one or more processing circuits, such as a processor, and / or one or more software modules for performing the method and / or actions, as processing modules 801.

[0108] The image sensor 800 may further include a memory 802 which may contain the computer program 803, such as by housing or storing the computer program 803. The computer program 803 includes “instructions” or “code” that can be directly or indirectly executed by the image sensor 800 to perform the methods and / or actions. The memory 802 may include one or more memory units which may further be arranged to store data such as configurations, data, and / or values ​​that are involved in or for performing the functions and actions of the embodiments herein.

[0109] Furthermore, the image sensor 800 may include, as an example of a hardware module, a processing circuit 804 involved in processing and, for example, encoding data, and may include or correspond to one or more processors or processing circuits. The processing module 801 may include the processing circuit 804, and may, for example, be "embodied in the form of" the processing circuit 804 or "realized by" the processing circuit 804. In these embodiments, the memory 802 may include a computer program 803 executable by the processing circuit 804, thereby enabling or configuring the image sensor 800 to operate to perform the methods and / or actions described above.

[0110] The image sensor 800, for example, the processing module 801, may include an input / output (I / O) module 805 configured to engage in any communication to and from other units and / or devices, such as sending and / or receiving information to and from other devices, for example. The I / O module 805 may be exemplified, where applicable, by an acquisition module, for example, a receiving module, and / or a providing module, for example, a transmitting module.

[0111] In addition, the image sensor 800 typically includes a sensing circuit 806 as an example of a hardware module involved in sensing light and / or images, which fully or partially corresponds to or fully includes those discussed above with respect to the image sensor 331, including an image sensing area such as the image sensing area 351.

[0112] Furthermore, in some embodiments, the image sensor 800, for example, the processing module 801, includes one or more of the acquisition module, determination module, exposure module, and providing module, as examples of hardware and / or software modules for performing the actions of the embodiments herein. These modules may be fully or partially implemented by the processing circuit 804.

[0113] therefore, The image sensor 800, and / or processing module 801, and / or processing circuit 804, and / or decision module are operable or configured to determine the second ROI, ROI2, in accordance with the predetermined relationship that ROI2 has with respect to the first ROI, ROI1.

[0114] The image sensor 800, and / or processing module 801, and / or processing circuit 804, and / or exposure module are operable or configured to expose at least the image sensor pixels of ROI2 to a second exposure, thereby causing the at least the image sensor pixels of ROI2 to acquire the second pixel value.

[0115] The image sensor 800, and / or processing module 801, and / or processing circuit 804, and / or I / O module 805, and / or providing module are operable or configured to use ROI2 to provide the second pixel values ​​of the image sensor pixels of ROI2 for the further processing by the image sensor.

[0116] Figure 9 is a flowchart illustrating a second method according to embodiments of this specification, which has been shown above but is disclosed in some more detail here. The method is preferably performed by a 3D imaging system based on optical triangulation, such as imaging system 605. The following actions, which may form the method, are for reading data from such an imaging system, or in other words, relating to data reading. Hereinafter, imaging system 605 is used as a non-limiting example of an imaging system. Imaging system 605 is for 3D imaging of an object, for example, object 620, and includes a camera, such as camera 630, having an image sensor, for example, one of image sensors 331, 631, 800, and one or more light sources, such as light sources 610, 640, for providing one or more illumination of object 620 during imaging. In other words, the image sensor should be an image sensor configured to work according to some embodiments of this specification, as described above. Hereinafter, for ease of understanding, the image sensor and parts thereof are referred to using the same illustrative reference numbers used above with respect to the image sensor.

[0117] Please note that the following actions may be performed in any preferred order, and / or may overlap completely or partially in time, where preferable.

[0118] Action 901 The imaging system 605 performs a first readout of first data from the image sensor 631. The first data is based on the first pixel value of an image sensor pixel of a first ROI, ROI1, for example, ROI1 365. The first pixel value is generated as a result of the operation of the imaging system 605, including the image sensor 631, such that the first exposure of the image sensor was under first illumination of an object 620 provided by one or more light sources, for example, from light source 610.

[0119] Action 902 The imaging system 605 performs a second readout of second data from the image sensor 631. The second data is based on the second pixel value of an image sensor pixel of a second ROI, ROI2, for example, ROI2 465. The second pixel value is generated as a result of the operation of the imaging system 605, including the image sensor, such that the second exposure of the image sensor was under second illumination of object 620 provided by one or more light sources, for example from light source 640.

[0120] As should be understood from the above introductory discussion relating to embodiments of this specification, the second illumination is preferably different from the first illumination. However, in some embodiments, the same illumination may be used for 3D imaging and then for 2D imaging as a different, separate imaging.

[0121] Furthermore, the first illumination includes light used for 3D imaging based on optical triangulation. Thus, the first illumination may be light conventionally used for 3D imaging based on optical triangulation, such as light planes or light edges projected as lines of light onto object 620, as described above with respect to such systems. The light stabilizing the first illumination, such as light 611, may therefore be light from a laser or LED, such as a high-brightness LED including a micro-LED.

[0122] The second illumination may include light for providing 2D related information about object 620 based on the surface reflection of the second illumination from the object being captured by camera 630 and image sensor, thereby so that the 2D related information about object 620 is included in the second pixel value.

[0123] Figure 10 is a schematic block diagram illustrating embodiments of how device 1000 may be configured to control the 3D imaging system 605 to perform the second method and / or action described above in relation to Figure 9, and / or to cause the 3D imaging system 605 to perform the second method and / or action described above in relation to Figure 9, i.e., device 1000, part of the imaging system, and how device 1000 may be configured to control the 3D imaging system 605 to perform the second method and / or action described above in relation to Figure 9.

[0124] For example, device 1000 may include or correspond to camera 630 and / or computing device 633, but as should be understood, two or more devices may typically provide functions relevant when an imaging system such as 3D imaging system 605 is operated and may be involved in controlling the imaging system to operate the imaging system as described above in relation to Figure 9, so it is not limited to a single device.

[0125] Device 1000 may include processing means, such as one or more hardware modules including one or more processing circuits, such as a processor, and / or one or more software modules for performing the method and / or actions, such as a processing module 1001.

[0126] Device 1000 may further include memory 1002 which may contain or store computer program 1003. Computer program 1003 includes “instructions” or “codes” that can be executed directly or indirectly by device 1000, respectively, to perform the methods and / or actions described herein. Memory 1002 may include one or more memory units which may further be arranged to store data such as configurations, data, and / or values ​​that are involved in or for performing the functions and actions of the embodiments herein.

[0127] Furthermore, each device 1000 may include, as an example of a hardware module, a processing circuit 1004 involved in processing and, for example, encoding data, and may include or correspond to one or more processors or processing circuits. A processing module 1001 may include such a processing circuit 1004, for example, "embodied in the form of" such a processing circuit 1004 or "realized by" such a processing circuit 1004. In these embodiments, a memory 1002 may include a computer program 1003 that can be executed by each of the processing circuits 1004, thereby enabling or configuring each device 1000 to operate to perform the method and / or actions.

[0128] Generally, device 1000, for example, processing module 1001, includes input / output (I / O) modules 1005 configured to engage in any communication to and from other units and / or devices, such as sending and / or receiving information to and from other devices, for example by performing such actions. The I / O modules 1005 may be exemplified, where applicable, by acquisition, for example, a receiving module, and / or provision, for example, a transmitting module.

[0129] Furthermore, in some embodiments, the device 1000, for example, the processing module 1001, includes one or more of the acquisition module, control module, execution module, and exposure module, as examples of hardware and / or software modules for performing the actions of the embodiments herein. These modules may be fully or partially implemented by the processing circuit 1004.

[0130] therefore, Device 1000, and / or processing module 1001, and / or processing circuit 1004, and / or I / O module 1005, and / or execution module are operable or configured to perform the first readout of first data from the image sensor and the second readout of second data from the image sensor.

[0131] Figure 11 is a schematic diagram showing several embodiments relating to computer programs 803 and / or 1003 and their carriers for causing the image sensor 800 and / or device 1000 discussed above to perform the method and actions or to cause the method and actions to be performed.

[0132] Each computer program 803, 1003 includes instructions that cause the image sensor 800 and / or 3D imaging system 605 to operate as described above when executed by a suitable processing circuit, such as processing circuits 804, 1004, and / or processing modules, such as processing modules 801, 1001. In some embodiments, one or more carriers are provided, namely carriers containing computer programs, or more specifically, data carriers such as computer program products. Each carrier may be one of electronic signals, optical signals, radio signals, and computer-readable storage media, such as the computer-readable storage media 1101 schematically shown in the figure. Thus, computer programs may be stored in the computer-readable storage media 1101. Depending on the carrier, transient propagating signals may be excluded, and accordingly, the data carrier may be called a non-transient data carrier. Non-limiting examples of data carriers that are computer-readable storage media include memory cards or memory sticks, disk storage media, or mass storage devices generally based on hard drives or solid-state drives (SSDs). The computer-readable storage medium 1101 may be used to store data accessible via a computer network 1102, such as the Internet or a local area network (LAN). The computer program may also be provided as a pure computer program or contained in one or more files. One or more files may be stored on the computer-readable storage medium 1101 and made available, for example, by download via the computer network 1102, as shown in the figure, such as via a server. The server may be a web or file transfer protocol (FTP) based server, etc.One or more files may be executable files to be downloaded directly or indirectly to the image sensor 800 and / or the device 1000 and executed on the image sensor 800 and / or the device, for example, by execution by a suitable respective processing circuit. One or more files may also be for intermediate download and compilation, including the same or a different processor, to make those files executable before further download and execution that causes the image sensor 800 and / or imaging system 605 to work as described above.

[0133] It should be noted that any of the aforementioned processing modules and circuits may be implemented as software and / or hardware modules, for example, on existing hardware and / or as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. It should also be noted that any of the aforementioned hardware modules and / or circuits may be distributed across several separate hardware components, whether, for example, they are contained in a single ASIC or FPGA, individually packaged, or assembled into a system-on-a-chip (SoC).

[0134] Furthermore, those skilled in the art will understand that the modules and circuits discussed herein may refer to hardware modules, software modules, analog and digital circuits, and / or a combination of one or more processors configured to perform the above-described methods and actions when executed by these modules, and / or one or more processors, for example, using software and / or firmware stored in memory, which may cause the above-described methods and actions to be performed by the devices, sensors, etc.

[0135] Identification by any identifier in this specification may be implicit or explicit. Identification may be unique, for example, in a specific context relating to a particular computer program or program provider.

[0136] As used herein, the term “memory” may refer to data memory for storing digital information, typically such as hard disks, magnetic storage, media, portable computer diskettes or disks, flash memory, random access memory (RAM), etc. Furthermore, memory may also refer to the internal register memory of a processor.

[0137] It should be noted that all enumerative terms, such as the first device, the second device, the first surface, the second surface, etc., should therefore be considered non-restrictive, and the terms themselves do not suggest any particular hierarchical relationship. On the contrary, unless there is some explicit information, enumerative naming should be considered simply a way of realizing different names.

[0138] When used herein, the expression "configured to..." may mean that a processing circuit is configured or adapted by software or hardware configuration to perform one or more of the actions described herein.

[0139] As used herein, the terms “number” or “value” may refer to any type of digit, such as a binary number, a real number, an imaginary number, or a rational number. Furthermore, a “number” or “value” may be one or more characters, such as a letter or string of characters. Also, a “number” or “value” may be represented by a bit string.

[0140] When used herein, the expressions “may” and “in some embodiments” are generally used to indicate that the described features may be combined with any other embodiments disclosed herein.

[0141] Features that may be present in only some embodiments are typically depicted using dotted or dashed lines in the drawings.

[0142] When the words "comprise" or "comprising" are used, they are interpreted as non-restrictive, meaning "consist at least of".

[0143] The embodiments described herein are not limited to those described above. Various alternatives, modifications, and equivalents may be used. Accordingly, the embodiments described above should not be construed as limiting the scope of the disclosure as defined by the appended claims. [Explanation of symbols]

[0144] 105 Imaging System 110 Light source 111 Light plane 112 Light Lines 120 The first object 121 The second object 122 Conveyor Belt 123 Coordinate System 130 Cameras 141-1~141-N Profile Images 143 Coordinate System 205 Imaging System 210 First light source 211 First light, light plane 220 Object 230 Cameras 231 Image Sensor 232 Field of view 233 Computing Devices 240 Second light source 241 The Second Light 331 Image Sensor 351 Image Sensing Area 352, 352-1~352-M columns Lines 354-1~352-12 356 M parallel ADCs 357 Calculation circuit 358 I / O circuit 359 Control circuit 361 Intensity Peak Line Height of ROI 362, Height of ROI 1 363 WAM, ROI 363' ROI 363a WAM lower edge line 363b WAM upper line 364 Light Distribution 365 ROI1 365' ROI1 463a Second ROI 463a' Second ROI 463b Second ROI 463b' Second ROI 463c Second ROI 463c' Second ROI 463d Second ROI 463d' Second ROI 465a~d ROI2 465a~d' ROI2 467 Offset 565 Second ROI, ROI2 605 3D Imaging System 610 First light source 611 The First Light 620 Object 630 Camera 631 Image Sensor 632 field of view 633 Computing Devices 640 Second light source 641 The Second Light 800 Image Sensor 801 Processing Module 802 memory 803 Computer Programs 804 Processing Circuit 805 I / O module 806 sensing circuit 1000 devices 1001 Processing Module 1002 memory 1003 Computer Programs 1004 Processing Circuit 1005 I / O Module 1101 Computer-readable storage media 1102 Computer Networks

Claims

1. A method for providing pixel values ​​of image sensor pixels in a second ROI (465, 465', 565) based on a first ROI (365, 365') performed by an image sensor (331, 631, 800), A method comprising: the first ROI (365, 365') being used by the image sensor (331, 631, 800) to provide a first pixel value of the image sensor pixel of the first ROI (365, 365') resulting from a first exposure of the image sensor pixel of at least the first ROI, wherein each of the ROIs partially covers the entire image sensor pixel, thereby partially covering the image sensing area (351) of the image sensor (331, 631, 800) including the entire image sensor pixel; - A step (702) of determining the second ROI (465, 465', 565) according to a predetermined relationship that the second ROI (465, 465', 565) has with respect to the first ROI (365, 365'), - Step (703) of exposing at least the image sensor pixels of the second ROI (465, 465', 565) to a second exposure, thereby causing at least the image sensor pixels of the second ROI (465, 465', 565) to acquire a second pixel value, - Step (704) of providing a second pixel value of the image sensor pixels of the second ROI (465, 465', 565) for further processing by the image sensor (331, 631, 800) using the second ROI (465, 465', 565) and Methods that include...

2. The method according to claim 1, wherein the predetermined relationship is such that the second ROI (465, 465', 565) can be shaped like the first ROI (365, 365'), but is positioned on the image sensing area (351) with a specific offset relative to the first ROI (365, 365'), thereby covering at least some image sensor pixels different from those covered by the first ROI.

3. The method according to claim 1 or 2, wherein the predetermined relationship determines a shape on the image sensing area (351) having a specific reduced or increased spread along the specific direction such that the second ROI (465, 465', 565) has a specific reduced or increased spread along the specific direction such that the second ROI (465, 465', 565) covers more or fewer pixels along the specific direction than the first ROI (365, 365').

4. The method according to claim 1, wherein the predetermined relationship is that the second ROI (465, 465', 565) is a copy of the first ROI, and thereby covers the same image sensor pixels as the first ROI (365, 365').

5. The method according to any one of claims 1 to 4, wherein the first ROI (365, 365') is determined by the image sensor (331, 631, 800) to cover the image sensor pixels based on a first pixel value of the image sensor pixels.

6. The method according to any one of claims 1 to 5, wherein the first ROI (365, 365') is determined by the image sensor (331, 631, 800) to cover the respective intensity peaks resulting from the first exposure along each pixel line of the image sensing area (351).

7. The method according to claim 6, wherein the first ROI (365, 365') is determined by the image sensor (331, 631, 800) to cover the same number of pixels of each pixel line around the respective intensity peak of each pixel line.

8. In an image sensor (331, 631, 800) for providing the pixel values ​​of image sensor pixels in a second ROI (465, 465', 565) based on a first ROI (365, 365'), The first ROI (365, 365') is used by the image sensor (331, 631, 800) to provide a first pixel value of the image sensor pixel of the first ROI (365, 365') resulting from at least a first exposure of the image sensor pixel of the first ROI, wherein each of the ROIs partially covers the entire image sensor pixel, thereby partially covering the image sensing area (351) of the image sensor (331, 631, 800) including the entire image sensor pixel, The second ROI (465, 465', 565) is determined according to a predetermined relationship that the second ROI (465, 465', 565) has with respect to the first ROI (365, 365') (702), Exposing at least the image sensor pixels of the second ROI (465, 465', 565) to a second exposure (703), thereby exposing at least the image sensor pixels of the second ROI (465, 465', 565) to a second pixel value, and Using the second ROI (465, 465', 565), a second pixel value of the image sensor pixels of the second ROI (465, 465', 565) is provided for further processing by the image sensor (331, 631, 800) (704). Image sensors (331, 631, 800) are configured to perform the following actions.

9. A method for reading data from a three-dimensional (3D) imaging system (605) based on optical triangulation, The imaging system (605) is for 3D imaging of an object (620) and includes a camera (630) having an image sensor (331, 631, 800) as described in claim 8, and one or more light sources (610, 640) for providing one or more illumination of the object (620) during imaging, - Step (901) of performing a first readout of first data from the image sensors (331, 631, 800), wherein the first data is based on the first pixel value of the image sensor pixels in the first ROI (365, 365'), and the first pixel value is a result of the operation of the imaging system (605) including the image sensors (331, 631, 800), such that the first exposure of the image sensors (331, 631, 800) was under first illumination of the object (620) provided by one or more light sources (610, 640); - Step (902) of performing a second readout of second data from the image sensors (331, 631, 800), wherein the second data is based on the second pixel value of the image sensor pixels of the second ROI (465, 465', 565), and the second pixel value is a result of the operation of the imaging system (605) including the image sensors (331, 631, 800), such that the second exposure of the image sensors (331, 631, 800) was under second illumination of the object (620) provided by one or more light sources (610, 640) and Methods that include...

10. The method according to claim 9, wherein the second illumination is different from the first illumination.

11. The method according to any one of claims 9 to 10, wherein the first illumination includes light used for 3D imaging based on optical triangulation.

12. The method according to any one of claims 9 to 11, wherein the second illumination includes light for providing two-dimensional (2D) related information about the object (620) based on the surface reflection of the second illumination from the object being captured by the camera (630) and the image sensors (331, 631, 800), so that the 2D related information about the object (620) is included in the second pixel value.

13. An imaging system (605) for three-dimensional (3D) imaging of an object (620) based on optical triangulation, for providing a second readout of a second pixel value in relation to a first readout of a first pixel value, comprising a camera (630) having an image sensor (331, 631, 800) as described in claim 8, and one or more light sources (610, 640) for providing one or more illumination of the object (620), (901) perform a first readout of first data from the image sensor (331, 631, 800), wherein the first data is based on the first pixel value of the image sensor pixel of the first ROI (365, 365'), and the first pixel value is a result of the operation of the imaging system (605) including the image sensor (331, 631, 800), such that the first exposure of the image sensor (331, 631, 800) was under first illumination of the object (620) provided by one or more light sources (610, 640), and (902) Performing a second readout of second data from the image sensors (331, 631, 800), wherein the second data is based on the second pixel value of the image sensor pixels of the second ROI (465, 465', 565), and the second pixel value is a result of the operation of the imaging system (605) including the image sensors (331, 631, 800), such that the second exposure of the image sensors (331, 631, 800) was under second illumination of the object (620) provided by one or more light sources (610, 640). An imaging system (605) configured to perform the following.

14. One or more computer programs (803, 1003) that, when executed by one or more processors, include instructions causing the image sensor (331, 631, 800) according to claim 8 and / or the imaging system (605) according to claim 13 to perform the method according to any one of claims 1 to 7 and / or the method according to any one of claims 9 to 12.

15. One or more carriers comprising one or more computer programs (803, 1003) according to claim 14, wherein one or more carriers are, namely, one or more of electronic signals, optical signals, wireless signals, or computer-readable storage media (1101).

Images