Image recording methods

Abstract

An image recording method involves using a camera, which includes an image sensor with a multitude of light-sensitive detector elements, to record a sequence of individual images. Each image contains a multitude of pixels, each corresponding to one of the detector elements. During recording, an individual analysis is performed in the camera for at least a subset of the individual images in each sequence. The result of this analysis is the assignment of a defect class to each pixel in the respective image. Based on the results of these individual analyses, a final overall analysis is performed, which assigns a defect class to each detector element.

Description

[0001] The invention relates to an image recording method, in particular a method for recording moving image data using a camera, wherein the moving image data comprises at least one image sequence, i.e., a sequence of individual images. In addition, the moving image data can also comprise metadata associated with the respective image sequence.

[0002] To record individual images, a modern camera typically includes an image sensor with a plurality of light-sensitive detector elements, also referred to as pixels of the image sensor (sensor pixels), which are designed to generate electrical charges in response to incident light. The electrical charge generated by each detector element can then be converted into an image signal value of a pixel corresponding to the detector element in the respective individual image. Together, the image signal values ​​then form the individual image, whose pixels are also referred to as pixels (image pixels).

[0003] Image sensors in typical motion picture cameras comprise several million detector elements. Due to production reasons, it is generally unavoidable that at least some of these detector elements, and possibly also one or more smaller groups of adjacent detector elements (pixel clusters), are defective (so-called pixel errors). Such defective detector elements result in image signal values ​​that deviate significantly from the average of the (largely consistent) image signal values ​​produced by the remaining, intact detector elements under the same exposure. In particular, the image signal values ​​of defective detector elements can be significantly smaller or significantly larger, so that pixels in the respective individual image that correspond to a defective detector element appear excessively dark or excessively bright in the image.

[0004] Such faulty image signal values ​​do not contain any usable image information, which is why they are usually discarded to correct the affected pixels and replaced with values ​​obtained by interpolating non-faulty image signal values ​​from pixels in the vicinity (local neighborhood) of the respective affected pixel. However, this requires knowing which detector elements are defective, so that the image signal values ​​originating from them are unusable.

[0005] Permanently defective detector elements (static pixel errors) can be relatively easily identified, for example, through a calibration process to which the image sensor is subjected. The image sensor is illuminated as uniformly as possible so that each of the detector elements is illuminated with at least essentially the same intensity of light. For intact detector elements, it can therefore be expected that they each generate essentially the same electrical charge, which is then converted into an image signal value that is essentially the same for all detector elements. These detector elements can be identified as defective if the respective image signal value for individual detector elements or groups of detector elements deviates significantly from the image signal value that is essentially uniform for the other detector elements.Information about the positions of detector elements identified as defective can therefore be available before a respective image sequence is recorded and then taken into account immediately during or after recording, or at least retrospectively, to correct the recorded individual images. Furthermore, static pixel errors can also be identified retrospectively—either through subsequent (re)calibration (e.g., during camera maintenance) or based on previously recorded image data—and then appropriately corrected.

[0006] However, the treatment of dynamic defects in which the affected detector element does not permanently but only sporadically or intermittently leads to a respective faulty image signal value, i.e. to an image signal value that deviates from the image signal value that an intact detector element would deliver, is significantly more difficult. Such dynamic pixel defects can, for example, be caused by a high temperature of the camera, in particular of the image sensor, and can therefore depend on the current thermal state of the camera (or also on other, in particular physical, environmental conditions such as pressure and / or humidity) during the respective recording. However, dynamic pixel defects can also have a temporal cause and depend, for example, on the age of the camera or the image sensor, and become more likely with increasing age.Furthermore, the respective image environment of a pixel can also influence its functionality, for example due to crosstalk. For example, a dynamic pixel error can occur depending on the spectrum and / or intensity of the light incident on the respective pixel and its surrounding pixels. Correcting dynamic pixel errors therefore advantageously has a direct and precise relationship to the ambient conditions, the time, and the image content of the respective recording.

[0007] However, dynamic pixel defects cannot be reliably detected during a calibration process and are therefore usually not known in advance. They can only be detected after recording within the affected individual images. Algorithms are known for this purpose that compare the pixels of a given individual image with pixels from the local neighborhood and infer the presence of a defect from significant deviations, possibly taking local spatial structures (e.g., the course of lines or edges) into account.

[0008] However, a strong deviation between the image signal value of a particular pixel or the image signal values ​​of a particular group of pixels and the image signal values ​​of neighboring pixels does not necessarily have to be due to a defect, but can also be caused by the image content, which may not be able to be distinguished algorithmically, or only with great difficulty. For example, it is particularly challenging to distinguish dynamic pixel errors, which lead to excessively bright pixels in the respective individual image, from real small highlights, which appear, for example, in people's eyes or on jewelry. Sometimes, during the recording of an image sequence, the light is deliberately directed so that such highlights appear because they make the image sequence appear more vivid. However, the highlights are sometimes so small that a distinction cannot be made between defective pixels or defective pixel clusters.If a pixel detection algorithm classifies pixels forming a highlight as defective, and the highlight is subsequently removed or altered, the intentionally applied effect is disrupted. In an image sequence, there is also the risk that such a disturbance will be introduced intermittently into the sequence of individual images. The highlight would then flash or change color.

[0009] Particularly in the creative field, for example, in film production, algorithms for detecting pixel defects, especially dynamic pixel defects, must therefore operate rather conservatively to avoid the generation of disturbing image artifacts. As a result, not all pixel defects are reliably detected algorithmically, which leads to additional effort in post-processing the respective image sequence, as the undetected pixel defects ultimately have to be removed by manual retouching.

[0010] The reliability of the algorithmic identification of dynamic pixel defects can be significantly increased if not just a single image of the image sequence is analyzed, but the entire image sequence is analyzed over time. This is because a truly defective detector element causes noticeable deviations in the resulting image signal values, not necessarily in every single image of a respective image sequence, but always at the same location (namely, at the pixel of the affected individual images corresponding to the detector element), which is independent of the image content, which changes over time. The fact that a pixel defect does not change its position can be used to distinguish between a defective pixel and real image content (such as a small highlight).In principle, it is therefore advantageous for the correction of pixel errors if the image sequence is analyzed as a whole in order to identify the pixel errors, especially the dynamic pixel errors.

[0011] However, this is problematic because the camera must be available for recording further image sequences as quickly as possible after recording an image sequence, and must also be able to output or transmit the recorded image sequence as quickly as possible. This leaves little time after recording to examine the recorded image sequence as a whole for dynamic pixel errors without delaying further recordings. However, even with post-processing of the recorded image sequence outside the camera, for example, using an external image processing device, analyzing the image sequence as a whole is not without its problems.Software for image processing of moving image data is required to enable one or more individual images of the image sequence to be processed in real time from an arbitrarily selected starting point within an image sequence, so that here too there is usually no time to first analyze the image sequence as a whole with regard to dynamic pixel errors without thereby delaying the image processing.

[0012] It is an object of the invention to provide an image recording method which enables particularly reliable identification, in particular of dynamic pixel errors, and which can be integrated into existing processes in such a way that a delay in these processes is largely avoided.

[0013] The object is achieved by an image recording method having the features of claim 1. Advantageous embodiments emerge from the subclaims, the present description and the figure.

[0014] The image recording method according to the invention comprises recording a sequence of individual images by means of a camera which comprises an image sensor with a plurality of light-sensitive detector elements, each of which comprises a plurality of pixels, each of which corresponds to a respective one of the detector elements.

[0015] The camera's image sensor comprises a plurality of light-sensitive detector elements, preferably arranged in rows and columns. The detector elements form pixels of the image sensor and are identifiable within the image sensor, particularly by their respective position (row and column). Each of the detector elements can expediently be configured to generate a respective electrical charge depending on the intensity of light incident on the respective detector element.

[0016] The recording of a respective individual image can comprise converting the charges generated by the detector elements during a respective individual exposure (i.e. the time period from the erasure of a charge storage device assigned to the respective detector element, during which charge is collected in the charge storage device, until the time at which the collected charge is read out; so-called integration time) into a respective image signal value. This can comprise, in particular, converting the generated charges into a voltage proportional to the respective charge, and converting the resulting analog voltage values, optionally after they have been amplified (for example by means of a transistor circuit), into an image signal value (for example by means of an A / D converter). The totality of the image signal values ​​thus obtained then forms - optionally modified (e.g.The respective individual image is then supplemented with additional image signal values ​​derived from these (e.g., for the purpose of white balance or gamma correction) and / or with additional image signal values ​​derived from them (e.g., for the purpose of so-called demosaicing). Therefore, the respective individual image does not necessarily have to be raw image data, although this is fundamentally possible; it is also conceivable that the individual images have already been processed to a certain extent, even before the analyses and / or corrections explained below are applied.

[0017] Each image signal value is assigned to a respective one of the aforementioned pixels of the individual image, namely, expediently, the pixel corresponding to the detector element that generated the electrical charge on which the respective image signal value is based. A pixel can correspond to a detector element, in particular, insofar as the position of the pixel within the respective individual image corresponds to the position of the detector element within the image sensor. The pixels, which, like the detector elements, are preferably arranged in rows and columns, thus form pixels of the respective individual image and are identifiable within the individual image, in particular by their respective position (row and column).

[0018] The recorded individual images are still images. The camera records a sequence of such individual images, i.e., an image sequence. The individual images are preferably recorded at regular intervals so that the sequence has a constant frame rate, which can be, for example, 24, 25, 30, 50, or 60 fps (frames per second). The serial playback of the sequence of individual images creates the impression of movement. In this respect, the camera is a motion picture camera.

[0019] According to the invention, the image recording method further comprises that during the recording of the sequence for each individual image of at least a subset of the individual images of the sequence a respective individual analysis is carried out in the camera, as a result of which each pixel of the respective individual image is assigned a respective defect class.

[0020] At least some individual images from the recorded sequence of individual images are therefore individually subjected to a respective analysis. In particular, the individual analysis is limited to the respective individual image; it therefore does not take into account any individual images other than the respective individual image. The individual analysis can in particular comprise examining the respective individual image to determine whether one or more pixels in the individual image appear to be based on a defective detector element. For example, the individual analysis can comprise applying one of the algorithms mentioned above, by which the pixels in the respective individual image are each compared with pixels in the local vicinity; pixels or groups of pixels whose image signal values ​​deviate from those of neighboring pixels by a certain amount can then be classified as defective and assigned a corresponding defect class.Preferably, the assignment of defect classes to a respective pixel is carried out exclusively based on the image information contained in the respective individual image. However, it is also conceivable that the individual analysis includes additional information of a different kind, for example, the result of a calibration process for identifying static pixel defects (see above).

[0021] Fundamentally, the individual analysis aims to assign a respective defect class to each pixel of the respective individual image, i.e., to estimate for each pixel whether it represents real image content or is corrupted due to a defect in the detector element to which it corresponds. In other words, the individual analysis serves to identify pixel defects based on a respective individual image. For the image recording method according to the invention, the algorithm or principle used for the individual analysis is of secondary importance. In particular, the individual analysis can fundamentally be any method, possibly known per se, for identifying pixel defects in an individual image.In this respect, the image recording method according to the invention is independent of the analysis method specifically used for the individual analyses, provided that it is ensured that the respective analysis method makes it possible to assign a respective defect class to each pixel of the individual image for which the individual analysis is carried out.

[0022] Depending on the method used for the individual analysis, a total of just two or more defect classes may be provided, one of which can be assigned to each pixel. In principle, it is expedient for one of these defect classes to correspond to an intact pixel (intact detector element) and thus represent the normal case, so that pixels which, according to the assessment criteria of the respective method, do not appear to be corrupted due to a pixel defect are assigned to this defect class. Preferably, only one further defect class is provided, which then corresponds to a defective pixel and is assigned to all other pixels. In particular, to distinguish between different degrees of defect or defect probabilities, depending on the analysis method, several defect classes may also be provided, each corresponding to a (partially / potentially) defective pixel.

[0023] As a result, the individual analysis provides an assignment of a respective defect class to each pixel of the respective individual image. The assignment can, for example, be in the form of a corresponding table or, for each defect class, comprise a list of the pixels to which the respective defect class has been assigned. The assignment does not necessarily have to be explicit for all pixels or defect classes. In particular, the assignment can only be implicit for exactly one of the defect classes. For example, it can be provided that the result comprises an explicit assignment only for those pixels to which a defect class corresponding to a defective pixel is assigned, whereby the defect class corresponding to an intact pixel is only implicitly assigned to the remaining pixels, namely precisely because these pixels are not explicitly included in the result.For example, exactly two defect classes can be provided, namely one by which a respective pixel is classified as defective and one by which a respective pixel is classified as intact, whereby the result of the individual analysis only contains a list or table with the positions of the pixels classified as defective.

[0024] The individual analyses are performed during the recording of the sequence of individual images, i.e., for all individual images (with the possible exception of the last individual image or images of the sequence) before the recording of the sequence is completed. The individual analysis is preferably performed in real time so that it is reliably completed within a defined period of time, which can, in particular, follow at least substantially immediately after the recording of the respective individual image to which the individual analysis is applied.

[0025] For example, it is advantageous if the individual analysis of each individual image takes place in a period between the end of the recording of this individual image and the beginning of the recording of the subsequent individual image. However, the individual analysis can also overlap with the recording of one or more subsequent individual images. In particular, in the latter case, the individual analysis of each individual image can also overlap with the individual analysis of a subsequent (immediately or subsequently) individual image or even with the individual analyses of several subsequent individual images, provided the camera has the appropriate computing capacity.

[0026] If the time required for an individual analysis is greater than the inverse of the frame rate at which the sequence of individual images is recorded, a temporal overlap of the individual analyses (and the associated computational effort) can nevertheless be at least reduced or even completely avoided by performing the individual analysis not for each individual image of the entire sequence, but only for each individual image of a true subset of the individual images of the sequence. This subset can in particular correspond to a subsampling of the sequence. For example, the subset can comprise every nth individual image of the sequence, where n is a natural number greater than one. In particular, the subset can comprise every second, every third or every fourth individual image of the sequence. Preferably, the subset comprises at least every fifth individual image.The subset can therefore have an effective frame rate that is smaller than the frame rate of the entire sequence by a corresponding factor.

[0027] According to an advantageous embodiment, the individual analysis is completed within a time period from the end of recording of the respective individual image to which the individual analysis is applied, which time period is shorter than the time period between the end of recording of the respective individual image and the end of recording of the next respective individual image to which the individual analysis is applied. The said time period for performing each individual analysis is therefore at most as long as the inverse of the said effective frame rate; if the individual analysis is applied to each individual image of the entire sequence, the said time period is at most as long as the inverse of the frame rate at which the sequence is recorded.

[0028] According to the invention, the image recording method further comprises performing an overall analysis based on the results of the individual analyses, as the result of which each detector element is assigned a respective defect class. This overall analysis is preferably based on the results of all individual analyses performed for a respective individual image of the recorded sequence of individual images; thus, all individual analyses performed are incorporated into the overall analysis.

[0029] Since the overall analysis is based on the results of the individual analyses, it can be limited to evaluating the assignments of defect classes to the pixels of the individual images obtained as a whole. In particular, the overall analysis does not need to examine the individual images pixel by pixel for pixel defects. For this reason, the computational and, in particular, time expenditure for the overall analysis is significantly reduced compared to an analysis that also considers the entire sequence of individual images, but independently determines the desired information about defective pixels directly from the individual images and, moreover, only after the entire sequence has been recorded.

[0030] This method takes particular advantage of the fact that the pixels of each individual image correspond to a respective detector element of the image sensor, and thus pixels of different individual images (namely, those located at the same position within their respective individual image) correspond to the same detector element of the image sensor. Due to this correspondence, as a result of the overall analysis based on the results of the independent individual analyses, each detector element of the image sensor (or also each pixel of the image sensor or of a respective individual image) can be assigned a respective defect class.

[0031] The defect classes, each of which is assigned to the detector elements of the image sensor as a result of the overall analysis, are preferably the same defect classes as those assigned to the pixels of the respective individual image as a result of a respective individual analysis. In particular, as was the case for the assignments to the pixels as a result of a respective individual analysis, only two defect classes are preferably provided for the assignments to the detector elements as a result of the overall analysis, namely one that corresponds to an intact pixel (whose image signal values ​​can be directly used) and one that corresponds to a defective pixel (whose image signal values ​​cannot be directly used but require correction); however, more defect classes can also be provided, for example if static and dynamic pixel errors and / or different defect degrees or defect probabilities are to be differentiated.As with the individual analyses, the result of the overall analysis for one of the defect classes, in particular for the one corresponding to an intact pixel, may only implicitly contain its assignment to detector elements.

[0032] Preferably, the overall analysis is performed at least substantially immediately after the end of the recording of the sequence of individual images or, if it has already begun, is completed. This is because the overall analysis can begin before the last individual analysis has been performed, for example, even after the first individual analysis has been performed, and can then be updated, in particular continuously, based on one or more further individual analyses performed.

[0033] In principle, it is also conceivable that the overall analysis takes into account the results of individual analyses or overall analyses of one or more earlier recordings of a respective sequence of individual images. In this respect, the overall analysis can additionally (namely, in addition to being based on the results of the aforementioned individual analyses) be based on the results of earlier individual analyses that were carried out for individual images of one or more sequences of individual images that were recorded using the same camera before the aforementioned sequence of individual images was recorded. For example, the overall analysis can start from the result of an earlier overall analysis based on the results of these earlier individual analyses and then update this earlier overall analysis based on the results of the aforementioned (new) individual analyses.

[0034] In particular, according to a further advantageous embodiment, the method can further comprise storing the result of the overall analysis in a memory, in particular a memory of the camera, wherein the overall analysis, if the memory previously (before storing the result of the overall analysis) already contains the result of an earlier overall analysis (carried out during an earlier implementation of the method according to the invention using the same camera), is additionally based on the result of the earlier overall analysis. Carrying out the overall analysis can then comprise reading the result of the earlier overall analysis, if present, from the memory and taking it into account in the overall analysis alongside the results of the individual analyses, so that it is incorporated into the result of the overall analysis.When saving the result of the (current) overall analysis in the memory, the result of the earlier overall analysis can then be overwritten, which in this respect corresponds to an updating of the result in the memory. Since the result of the earlier overall analysis can in turn be based on the result of an even earlier overall analysis, etc., the result of the (current) overall analysis can be based on the results of all earlier overall analyses (all carried out during a respective earlier implementation of the method according to the invention using the same camera). The memory can always contain at least the most recent result, i.e. the result of the most recently carried out overall analysis (which also includes the results of all earlier overall analyses). In this way, the memory can always store up-to-date information about defective pixels, in particular about dynamic pixel errors.

[0035] The fact that the overall analysis is carried out based on the results of the individual analyses and, if applicable, also on the result of an earlier overall analysis does not preclude the overall analysis from being based on further information. In particular, the result of a calibration process for identifying static pixel defects (see above) can also be incorporated into the overall analysis. Furthermore, it can be provided that individual pixels can be selected (for example by a respective user of the camera) and assigned to a respective defect class (fundamentally independently of the aforementioned analyses), and that this information is also incorporated into the overall analysis. Such further information can be stored in the aforementioned memory, from which it can then be read out to carry out the overall analysis.The memory can thus represent a “learning” database, in which information from various sources that may be relevant for the assignment of a particular defect class to the detector elements and / or analysis results based on this information are stored and then made available as a basis for further analyses that may update previous results.

[0036] Regardless of whether the overall analysis is started before the end of the recording of the sequence of individual images or only after this, according to a further advantageous embodiment it is in any case preferred that the overall analysis is carried out (and also completed) while still in the camera. This ensures that the result of the overall analysis, i.e. the information about identified (dynamic) pixel errors, can be stored and / or output as part of the recorded image data together with the sequence of individual images and is thus immediately available for further processing of the image data, in particular for correcting the pixel errors. The correction of the pixel errors, on the other hand, is preferably not also carried out in the camera, but subsequently outside the camera, so that the camera can be quickly used again for further recordings.

[0037] According to a further advantageous embodiment, the overall analysis is based exclusively on the results of the individual analyses. In particular, the individual images and their image signal values ​​are not taken into account in the overall analysis; they therefore only flow into the overall analysis indirectly, namely via the results of the individual analyses. By limiting the data on the basis of which the overall analysis is carried out, it can advantageously be ensured that the overall analysis can be carried out particularly quickly, so that its result can be available almost immediately (for example, within a few seconds or even less time) after the end of the recording of the sequence of individual images. This can also be helped by the fact that the overall analysis can be based at least essentially on comparatively simple arithmetic operations, such as determining frequencies.

[0038] According to a further advantageous embodiment, the overall analysis comprises determining for each detector element and each defect class the frequency with which, as a result of a respective individual analysis, the respective defect class has been assigned to an image point corresponding to the respective detector element of the respective individual image to which the individual analysis has been applied, and that, as a result of the overall analysis, a respective defect class is assigned to each detector element depending on the determined frequencies.

[0039] In other words, according to this embodiment, the overall analysis simply counts how often a respective pixel has been classified as defective (or, in the case of more than two defect classes, e.g., as defective to a certain degree and / or with a certain probability) in the individual analyses. By determining such frequencies, pixels can be easily identified that repeatedly exhibit conspicuous image signal values, so that (the higher the frequency, the greater the probability) it can be assumed that the conspicuous image signal values ​​are not caused by real image information (such as a highlight), but are based on a dynamic pixel defect.

[0040] According to an advantageous development of this embodiment, the assignment of a respective defect class to a respective detector element depends on whether the frequency with which the respective defect class has been assigned to a pixel of the respective individual image corresponding to the respective detector element, to which the individual analysis has been applied, as a result of a respective individual analysis is greater than a predetermined or predeterminable threshold value. In particular, the assignment of a respective defect class to a respective detector element can depend exclusively on whether the said frequency is greater than the threshold value.

[0041] Such a threshold comparison is particularly low in complexity and can therefore be performed without significant computational effort or time. Simply put, the overall analysis classifies each pixel that has been classified as defective more frequently than a certain frequency (threshold) in the individual analyses as defective (i.e., it is assigned a defect class corresponding to a defective detector element).

[0042] The threshold value is either fixed as a constant in the overall analysis or can be specified, i.e. it can be adjusted by a user (of the camera), at least within a certain range, in order to adapt the sensitivity of the overall analysis to the respective circumstances.

[0043] The threshold can be defined absolutely, for example as a concrete number, or relatively.

[0044] For example, the threshold value can be defined relative to the number of individual images in the specified subset. If the individual analysis is performed for all individual images of the entire sequence of individual images, this number is equal to the total number of individual images in the sequence. Such a threshold value can be specified, for example, as a percentage. For example, it can be provided that, as a result of the overall analysis, every pixel that has been classified as defective by the respective individual analysis in at least 10%, in particular at least 20%, of the individual images for which an individual analysis has been performed is classified as defective overall.

[0045] Alternatively, the threshold value can be defined relative to an average frequency with which, as a result of a respective individual analysis, the respective defect class has been assigned to an image point corresponding to a respective detector element of the respective individual image to which the individual analysis has been applied. Such a threshold value can, for example, be specified as a factor. For example, it can be provided that, as a result of the overall analysis, each pixel which has been classified as defective at least twice, in particular three times, more often than all pixels on average by the individual analyses carried out is classified as defective overall. The average frequency can be defined as a simple average (in particular an arithmetic mean) of the frequencies with which the respective defect class has been assigned to a respective pixel by the individual analyses.However, the mean frequency can also be defined as a median, mode, or other mean value that is preferably suitable as an estimator of the expected value of a distribution. The factor can also be defined relative to a standard deviation of the frequencies.

[0046] According to a further advantageous embodiment, the method further comprises that the camera stores and / or outputs the result of the overall analysis, in particular in the form of metadata, together with the sequence of individual images.

[0047] Outputting can comprise any form of data transmission from the camera, for example, via a data output on the camera, where data can be transmitted wired or wirelessly to a device external to the camera, or by removing a data storage device from the camera on which data has been stored in the camera. Preferably, the sequence of individual images from the camera is stored or output exclusively together with the result of the overall analysis. In particular, it can be provided that the sequence of individual images can only be read from the camera once the overall analysis is complete.

[0048] The individual images can be written continuously to the camera's memory immediately after they have been recorded. Once the entire sequence has been recorded, the individual images can be combined into a moving image dataset, particularly in the form of a moving image file, which can have a (basically any) known video file format. The assignments of a respective defect class to the detector elements of the image sensor, resulting from the overall analysis, represent supplementary information about the sequence of individual images and can thus form metadata for the moving image dataset.

[0049] In principle, the sequence of individual images, particularly in the form of the aforementioned moving image file, and the result of the overall analysis can be saved or output together, but as two separate files (e.g. as a main file and a so-called sidecar file). Preferably, however, the sequence of individual images and the result of the overall analysis are saved or output as a single file. Depending on the format of the moving image file, the result of the overall analysis can be embedded as metadata in the moving image file in which the sequence of individual images is saved. Alternatively, the individual file can have a container format, in particular the Material Exchange Format (MXF) according to the SMPTE (Society of Motion Picture and Television Engineers) standard ST 377-1:2019, which contains the sequence of individual images on the one hand and the result of the overall analysis on the other, as well as any other data orMetadata are summarized as separate and fundamentally independent data sets.

[0050] Furthermore, it can be provided that, in addition to the result of the overall analysis together with the sequence of individual images, the camera also saves and / or outputs the results of the individual analyses, particularly in the form of additional metadata. The results of the individual analyses can be saved or output as a separate file in the same way as the result of the overall analysis, embedded in the respective moving image file, or combined with the moving image file as a single file in a container format.

[0051] According to a further advantageous embodiment, the method further comprises carrying out a correction of the sequence of individual images outside the camera, by which one or more pixels of one or more individual images of the sequence are modified depending on the defect class assigned to the detector element corresponding to the respective pixel according to the result of the overall analysis.

[0052] The aforementioned correction therefore depends on the result of the overall analysis. How the correction is actually carried out is of secondary importance. The invention is not limited to a specific correction method. Rather, within the scope of the inventive method, numerous different correction methods (be they comparatively simple or very complex) can be used for the aforementioned correction. In principle, the individual images can be present as raw image data when they are subjected to correction; however, they can also already be processed to a certain extent. The correction method applied can depend on the extent to which the individual images have already been processed, in particular on whether white balance, gamma correction and / or demosaicing has / have already been carried out.

[0053] For example, it can be provided that the correction only modifies those pixels of a respective individual image that correspond to a detector element of the camera's image sensor to which a defect class corresponding to a defective pixel has been assigned according to the result of the overall analysis, or whose respective value is based at least among other things (e.g. due to interpolation as part of demosaicing) on ​​the image signal value of such a detector element. Whether the respective pixel is actually modified can also depend on whether it even appears as a potential pixel defect within the respective individual image, for example because it differs noticeably from pixels in its local surroundings within the respective individual image. The correction can involve first determining this.For this purpose, the respective individual image can be analyzed as part of the correction, whereby this analysis can basically correspond to the individual analyses mentioned above.

[0054] If the correction requires not only the result of the overall analysis but also the results of the individual analyses performed in the camera, for example, because they were saved and / or output by the camera together with the sequence of individual images, the results of the individual analyses can also be used instead. The correction can then involve modifying a respective pixel of a respective individual image (at least or exactly) if it corresponds to a detector element that was classified as defective according to the result of the overall analysis and was also itself classified as defective according to the result of the individual analysis performed for the respective individual image.

[0055] Modifying a respective pixel may, in particular, involve replacing the image signal value of the respective pixel with a value interpolated from image signal values ​​of neighboring pixels. Various generally known interpolation methods, such as bilinear or bicubic interpolation, may be used.

[0056] Ultimately, the classification and correction of pixel defects is thus a two-part process. The pixels are classified particularly reliably directly in the camera because the image sequence is analyzed as a whole. This allows the influence of dynamic factors—such as time (age of the camera or image sensor), ambient conditions (especially the temperature of the camera or image sensor), and image content—of the respective recording to be taken into account, thus allowing dynamic pixel defects to be identified. At the same time, excessive computational effort is avoided by not applying this analysis to the entire image sequence after the image sequence has been recorded, but rather by analyzing the individual images (individual analyses), so that the final analysis (overall analysis) can be limited to evaluating the analyses of the individual images.This can be done relatively easily and quickly, so that the image sequence, along with information about the pixel classification, can be output almost immediately after the end of each recording, making the camera essentially immediately available for further recordings. The image sequence can then be corrected outside of the camera. Because the classification is already present along with the image sequence, excessive computational effort for analyzing the entire image sequence is avoided during correction, so that any individual image in the sequence can be selected essentially immediately during correction, and any pixel errors it contains can be corrected.

[0057] The invention is further explained below merely by way of example with reference to the figure, which schematically visualizes the sequence of the method according to the invention according to an exemplary embodiment.

[0058] The image recording method 11 illustrated in the figure is at least largely implemented in a camera 13, which is shown schematically in a highly simplified manner in the figure. The camera 13 comprises a lens 15, which images a scene (not shown) to be recorded located in front of the camera 13 onto an image sensor 17 of the camera.

[0059] The image sensor 17 comprises a plurality of light-sensitive detector elements 19 arranged in columns and rows, which generate electrical charges depending on the intensity of light from the scene that strikes them. The electrical charges generated by the detector elements 19 during a specific period of time in which light from the scene is transmitted to the image sensor 17 are (ultimately) converted into image signal values, which then together form an individual image 21 comprising a plurality of pixels 23 arranged in rows and columns, corresponding to the image sensor 17. Each image signal value is assigned to a respective one of the pixels 23, which in turn each correspond to a respective one of the detector elements 19, namely the detector element 19 whose position within the image sensor 17 corresponds to the position of the respective pixel 23 within the individual image 21.

[0060] According to the image recording method 11, a complete sequence 25 of individual images 23 is recorded by the camera 13. The individual images 23 are recorded at constant time intervals, so that the sequence 25 has a constant frame rate.

[0061] For the sake of clarity, only a single one of the detector elements 19 of the image sensor 17 is shown in the figure and, moreover, is depicted greatly enlarged (in this illustration, which is not to scale anyway). The image sensor 17 actually has several million such detector elements 19, which would therefore be too small to represent a single one. Likewise, of the pixels 23 of the individual images 21, each of which has a number of pixels 23 corresponding to the number of detector elements 19, only a single one, namely the pixel 23 corresponding to the detector element 19 shown, is shown and depicted greatly enlarged. Furthermore, only the first three and the last two individual images 21 of the sequence 25 are shown. However, the sequence 25 can in principle comprise any desired number of individual images 23.Furthermore, in the case of elements that are included multiple times in the figure and are shown identically to one another, only one of them is provided with a reference symbol in order to avoid a confusing representation.

[0062] According to the image recording method 11, while the sequence 25 is still being recorded in the camera 13, a respective individual analysis 27 is carried out in real time for each individual image 21 of the sequence 25, as the result of which analysis 29 a respective defect class is assigned to each pixel 23 of the respective individual image 21. (In principle, the individual analysis 27 could also be applied only to the individual images 21 of a true subset of the sequence 25; in the exemplary embodiment shown, however, it is applied to all individual images 21.) In particular, only two defect classes can be provided, namely one by which the respective pixel 23 is classified as corresponding to an intact detector element 19, and one by which the respective pixel 23 is classified as corresponding to a defective detector element 19. According to which criteria orThe algorithm according to which the pixels 23 are classified in the respective individual analysis 27 is of secondary importance, since in the image recording method 11 many different analysis methods are possible for the individual analysis 27.

[0063] In particular, the individual analysis 27 can be based at least substantially on comparing whether the pixels 23 of the respective individual image 21 have image signal values ​​that deviate excessively (beyond a certain degree) from the image signal values ​​of neighboring pixels 23 of the respective individual image 21. The rate at which the individual analyses 27 are performed preferably corresponds to the frame rate at which the individual images 21 are recorded. Each individual analysis 27 is preferably completed before the next individual analysis 27 begins, in particular before the next individual image 21 for which an individual analysis 27 is performed is recorded.

[0064] Furthermore, in the camera 13, an overall analysis 31 is then performed based on the results 29 of the individual analyses 27, as the result 33 of which a respective defect class is assigned to each detector element 19. Preferably, defect classes are provided that correspond to the defect classes provided in the individual analyses 27. As the result 33 of the overall analysis 31, each of the detector elements 19 can therefore be assigned, in particular, either a defect class that classifies the respective detector element 19 as intact, or a defect class that classifies the respective detector element 19 as defective.

[0065] The overall analysis 31 can be based at least essentially on the fact that for each detector element 19 of the image sensor 17, on the basis of the results 29 of the individual analyses 27, it is determined with which frequency in the individual images 21 the respective pixel 23 of the respective individual image 21 corresponding to this detector element 19 has been assigned a defect class that corresponds to a defective detector element 19, and that this detector element 19 is assigned a corresponding defect class if this frequency exceeds a threshold value.

[0066] The individual images 21 can be written continuously, in particular immediately after they have been recorded, to a memory (not shown) in the camera. After the last individual image 21 has been recorded, the entire sequence 25 of individual images 21 can then be stored as a single moving image file 35 in the camera 13. The result 33 of the overall analysis 31 can also be stored together with this as metadata—either embedded in the moving image file 35, combined with it in a container file, or as a separate file from the moving image file 35.

[0067] Subsequently, the moving image file 35 with the sequence 25 of the individual images 21 and the result 33 of the overall analysis 31 can be output together by the camera 13 and in particular transmitted directly or indirectly (via one or more further devices and / or at a later time) to an image processing device 37, where, based on the result 33 of the overall analysis 31, a correction of pixel errors in the sequence 25 of individual images 21 is carried out.

[0068] This correction comprises modifying pixels 23 of the individual images 21 that correspond to detector elements 19 that have been classified as defective according to the result 33 of the overall analysis 31. In other words, these pixels 23 are considered based on the result 33 of the overall analysis 31 to be based on a pixel defect and thus fundamentally in need of correction. The modification may, in particular, comprise replacing the image signal values ​​of these pixels 23 with a value determined by interpolating image signal values ​​of neighboring pixels 23 within the respective individual image 21.However, if a respective pixel 23 does not deviate excessively from neighboring pixels 23 (for example because it corresponds to a detector element 19 with a dynamic pixel error, but the dynamic pixel error happened not to have an effect on the recording of the individual image 21 containing the respective pixel 23), the modification may also be omitted, so that the correction does not necessarily modify all pixels 23 that correspond to detector elements 19 that have been classified as defective according to the result 33 of the overall analysis 31.

[0069] As a result, the image recording method 11 according to the invention provides moving image data which can be particularly reliably corrected with regard to possible, particularly dynamic, pixel errors in a simple manner, in particular with comparatively little computational effort and above all while avoiding excessive delays (especially in the camera, but also during further image processing), or which (if the method also includes the correction) have already been corrected. Reference symbol 11 Image recording methods 13 Camera 15 lens 17 image sensor 19 Detector element 21 single images 23 pixels 25 sequence of individual images 27 Individual analysis 29 Result of the individual analysis 31 Overall analysis 33 Result of the overall analysis 35 moving image files 37 Image processing device

Claims

[1] Image recording method (11), the method (11) comprising: - that by means of a camera (13) comprising an image sensor (17) with a plurality of light-sensitive detector elements (19), a sequence (25) of individual images (21) is recorded, each comprising a plurality of pixels (23), each of which corresponds to a respective one of the detector elements (19); - that during the recording of the sequence (25) for each individual image (21) of at least a subset of the individual images (21) of the sequence (25) a respective individual analysis (27) is carried out in the camera (13), as the result of which (29) a respective defect class is assigned to each pixel (23) of the respective individual image (21); - that based on the results (29) of the individual analyses (27) an overall analysis (31) is carried out, as the result of which (33) each detector element (19) is assigned a respective defect class. [2] An image recording method according to claim 1, wherein the individual analysis (27) is completed within a time period from the end of recording of the respective individual image (21) to which the individual analysis (27) is applied, which time period is shorter than the time period between the end of recording of the respective individual image (21) and the end of recording of the respective next individual image (21) to which the individual analysis (27) is applied. [3] Image recording method according to claim 1 or 2, wherein the method (11) further comprises that the result (33) of the overall analysis (31) is stored in a memory, in particular a memory of the camera (13), wherein the overall analysis (31), if the memory already contains the result of an earlier overall analysis, is additionally based on the result of the earlier overall analysis. [4] Image recording method according to one of the preceding claims, wherein the overall analysis (31) is carried out in the camera (13). [5] Image recording method according to one of the preceding claims, wherein the overall analysis (31) is based exclusively on the results (29) of the individual analyses (27). [6] Image recording method according to one of the preceding claims, wherein the overall analysis (31) comprises that for each detector element (19) and each defect class the frequency is determined with which, as a result (29) of a respective individual analysis (27), a pixel (23) of the respective individual image (21) corresponding to the respective detector element (19) to which the individual analysis (27) has been applied has been assigned the respective defect class, and that as a result (33) of the overall analysis (31), each detector element (19) is assigned a respective defect class depending on the determined frequencies. [7] Image recording method according to claim 6, wherein the assignment of a respective defect class to a respective detector element (19) depends on whether the frequency with which, as a result (29) of a respective individual analysis (27), the respective defect class has been assigned to an image point (23) of the respective individual image (21) corresponding to the respective detector element (19) to which the individual analysis (27) has been applied, is greater than a predetermined or predeterminable threshold value. [8] An image recording method according to claim 7, wherein the threshold value is defined relative to the number of individual images (21) in said subset. [9] Image recording method according to claim 7, wherein the threshold value is defined relative to an average frequency with which, as a result (29) of a respective individual analysis (27), the respective defect class has been assigned to a pixel (23) corresponding to a respective detector element (19) of the respective individual image (21) to which the individual analysis (27) has been applied. [10] Image recording method according to one of the preceding claims, wherein the method (11) further comprises that the camera (13) stores and / or outputs the result (33) of the overall analysis (31), in particular in the form of metadata, together with the sequence (25) of the individual images (21). [11] Image recording method according to one of the preceding claims, wherein the method (11) further comprises that outside the camera (13) a correction of the sequence (25) of individual images (21) is carried out, by means of which one or more pixels (23) of one or more individual images (21) of the sequence (25) are modified as a function of the defect class which is assigned to the detector element (19) corresponding to the respective pixel (23) according to the result (33) of the overall analysis (31).

Images