Image processing

Abstract

An image processing method comprises obtaining 204 a plurality of images, each of the plurality of images comprising a view of a scene and computing 206 a respective score for each of the plurality of

Description

Field 5

[001] The present application relates to image processing, and in particular to generating a new image of a scene. Background 10

[002] Many image processing techniques are known. One example is image suggestion where an image is output to a user based on it having a particular desired characteristic, such as low noise level, low blurring or some other quality. An image is typically selected based on a score that is computed to represent the extent to which an image meets the desired characteristic. Whilst such known techniques can be useful in terms of automatically outputting 15 an image in a time-saving manner, they are only able to select an image from amongst a set of prestored images. This can be problematic if the score of even the best image is relatively low or the selected image is unsuitable.

[003] Further, such conventional methods consume a lot of memory and use large models 20 to select the image. Thus, they are unsuitable and cannot run in real-time on constrained-resource devices, including mobile devices. Summary 25

[004] Embodiments can address one or more of the technical problems discussed above. Embodiments can provide an on-device, lightweight image assessment network that is compact enough for on-device operation. Embodiments can have the ability to generate a novel view of a scene with a desirable characteristic using, for example, neural radiance fields. Embodiments may use an efficient greedy algorithm that traverses the scene space by 30 following the gradient of a score to quickly find a point in the scene space that generates the image with the best score.

[005] According to a first aspect of the present invention, there is provided a computer-implemented image processing method comprising: 13 08 25 obtaining a plurality of images, each of the plurality of images comprising a view of a scene; computing a respective score for each of the plurality of images; 5 estimating a respective camera pose of each of the plurality of images; using the computed scores and the estimated camera poses to determine a new camera pose useable for generating a new image comprising a view of the scene and having 10 a score greater than a threshold, and generating the new image using the new camera pose, wherein the computed scores are computed using a scoring model comprising a 15 trained machine learning model, and the computed scores represent a characteristic of an image, and wherein the computed scores represent a characteristic comprising a level of noise of an image, a level of blurring of the image, or a quality of the image based on an extent to 20 which the image complies with universal photograph composition rules.

[006] The method may further comprise applying a pattern recognition process to the estimated camera poses and the computed scores to find a camera pose amongst the estimated camera poses associated with a high score amongst the computed scores. The 25 pattern recognition process may also identify an initial adjustment manner to be used in a step of seeking a candidate new camera pose. The camera pose found to have the high score may be selected as an initial camera pose for use in the step of seeking the new camera pose. If a camera score having a high score is not found then a random camera pose may be selected as the initial camera pose for use in the step of seeking the new camera pose. The initial 30 adjustment manner to be used in the step of seeking a candidate new camera pose may also be randomly selected in this case.

[007] The method may further comprise rendering an image using the initial camera pose. The rendered image may have a lower resolution than a resolution of the plurality of images. 35 The method may comprise computing a score for the low resolution image (typically using the scoring model), and determining whether the computed score of the low resolution image is 13 08 25 (equal to or) greater than a predetermined threshold. If the computed score of the low resolution image is (equal to or) greater than the predetermined threshold then the method may comprise outputting the camera pose used to generate the low resolution image as the new camera pose. If the computed score of the low resolution image is not equal to or greater 5 than the predetermined threshold then the method may comprise seeking a candidate new camera pose.

[008] The step of seeking the candidate new camera pose may comprise adjusting at least one component (e.g. x / y / z coordinate or rotation matrix) of the candidate new camera pose 10 based on whether an image generated using the candidate new camera pose having the at least one component adjusted improves or deteriorates the score of the generated image compared to a score of an image generated using a (previous) camera pose without the at least one component adjusted. 15

[009] The step of seeking the candidate new camera pose may comprise: adjusting a component of the candidate new camera pose at a current iteration of the method (including the step of seeking the candidate new camera pose) in a particular adjustment manner / action (e.g. increase or decrease; rotate clockwise or counter-clockwise) 20 to produce a modified camera pose; generating an image (typically a low resolution image) based on the modified camera pose; 25 computing a score for the image rendered based on the modified camera pose; determining whether the computed score is greater than a previous score computed for an image generated using a (previous or the initial) camera pose (having an un-adjusted component) during a previous iteration of the method; 30 if the computed score is greater than (or equal to) the previous score then adjusting the (same) component of the camera pose in the (same) particular adjustment manner at a next iteration of the method, and 35 if the computed score is not greater than (or equal to) the previous score then adjusting a different component of the camera pose in the particular adjustment manner at the next 13 08 25 iteration, or adjusting the (same) component of the camera pose in a different adjustment manner at the next iteration.

[010] The component may comprise: x-coordinate, y-coordinate orz-coordinate, ora rotation 5 matrix of the camera pose. The adjustment manner may comprise increasing or decreasing a value of the component or another adjustment, such as adjusting the value to represent a clockwise or counter-clockwise rotation of the camera pose. The adjustment manner may be selected from a set. A particular adjustment manner may be excluded for selection from the set if the computed score is not greater than (or equal to) the previous score. 10

[011] The method may comprise comparing the current iteration to an iteration threshold and if the current iteration is greater than (or equal to) the iteration threshold then outputting the modified camera pose as the new camera pose. 15

[012] The camera poses may be estimated using a camera pose estimating model, e.g. a trained machine learning model. The machine learning model may be trained using a plurality of images with known camera poses, and comparing the plurality of images with images rendered using the model. 20

[013] The step of generating the new image using the new camera pose may be performed using a machine learning model. The machine learning model may be trained using training data comprising camera poses and images to output an image rendered based on an input camera pose. The machine learning model may comprise a Neural Radiance Field (NeRF) model. 25

[014] The method may further comprise: determining whether at least one of the computed scores is greater than a score threshold, and 30 if at least one of the computed scores is greater than the score threshold then outputting the image of the plurality of images having the score greater than the score threshold, and 13 08 25 if at least one of the computed scores is not greater than the score threshold then performing the step of estimating the respective camera pose of each of the plurality of images. 5

[015] According to a further aspect of the present invention there is provided an apparatus adapted to perform image processing according to the first aspect.

[016] According to yet another aspect of the present invention, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, 10 causes the processor to carry out any of the methods described herein.

[017] The features described herein in relation to any aspect may apply equally to another aspect and therefore, for the sake of conciseness, are not repeated. 15

[018] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. 20

[019] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor 25 system, apparatus, or device, or any suitable combination of the foregoing.

[020] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code 30 components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs. 13 08 25

[021] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein. 5

[022] The techniques further provide processor control code to implement the above described methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD- 10 ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application 15 Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working 20 memory and program memory coupled to one or more of the components of the system.

[023] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such 25 logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media. 30

[024] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method. 13 08 25

[025] The methods described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model 5 may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and 10 performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[026] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile 15 memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a 20 plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a 25 desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / or may be implemented through a separate server / system.

[027] The Al model may consist of a plurality of neural network layers. Each layer has a 30 plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), 35 and deep Q-networks. 13 08 25

[028] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or 5 reinforcement learning. Brief description of drawings

[029] Implementations of the present techniques will now be described, by way of example 10 only, with reference to the accompanying drawings, in which:

[030] Figure 1 is a block diagram of a computing device configurable to execute embodiments; 15

[031] Figure 2 is a flowchart showing example steps performed by an embodiment;

[032] Figure 3 is a flowchart showing example steps of a new camera pose seeking process used by the embodiment; 20

[033] Figures 4A - 4C are diagrams showing examples of correlations between camera pose components and scores;

[034] Figure 5 is flowchart showing example camera pose component adjusting steps of the new camera pose seeking process, and 25

[035] Figures 6 includes graphs comparing performance of an embodiment of the new camera pose seeking process and a brute-force alternative. Detailed description of drawings 30

[036] Figure 1 is a block diagram of a computing device 100 configurable to execute embodiments of the invention. The device will normally comprise, or be associated with, at least one processor 102, memory 104 and a communications interface 106. The at least one processor 102 may comprise one or more of: a microprocessor, a microcontroller and an 35 integrated circuit. The memory 104 may comprise volatile memory, such as random access 13 08 25 memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example. The communications interface can provide data communication between the device and other devices / components, e.g. via a wireless internet 5 connection, a cellular network connection, or the like. The computing device may further include a user interface component interface 108, such as a touchscreen. Other components and features of the device, such as a housing, power source / supply, display, audio output, etc, will be well-known to the skilled person and need not be described herein in detail. 10

[037] In some embodiments the computing device 100 may comprise a constrained- resource device, but which has at least the minimum hardware capabilities required to use a trained neural network / ML model. The device may be: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a connected camera, etc. It will be understood that this is a non-exhaustive and non-limiting list of example devices. 15

[038] Figure 2 is a schematic illustration of an example method according to an embodiment and shows steps performed by means of software instructions being executed by the computing device 100. However, in some embodiments one or more of the steps may be performed by a remote computing device, such as a server or a cloud service, that is in 20 communication with the device 100. It will also be appreciated that at least one of the steps described herein may be re-ordered or omitted. One or more additional steps may be performed in some cases. Although the steps are shown as being performed in sequence in the Figures, in alternative embodiments some of them may be performed concurrently, possibly on different processors or cores. It will also be understood that embodiments can be 25 implemented using any suitable software, programming language, data editors, etc, and may be represented / stored / processed using any suitable data structures and formats.

[039] At step 202 the image processing method can be initiated. For example, the method may be initiated when an application, such as an image editor, executable by the computing 30 device 100 is opened by a user, or when a particular option in an application is selected. In some cases, the user may be prompted by the application to capture several images of a scene, e.g. different views of the same scene from different angles and / or camera positions, using a still or video camera of the computing device. This can provide a plurality of images relating to the scene that are saved to the memory 104 and which can be obtained for 35 processing at step 204. 13 08 25

[040] In other cases the method may obtain a plurality of images related to a scene at step 204 by retrieving them from a data store or a remote device that is in communication with the computing device 100, e.g. based on user selection of a thumbnail image of the scene. Each image will comprise a view (e.g. from a particular location, angle, etc) of the same scene. In 5 cases where the images comprise photographs they will typically have been taken during the same short time period, e.g. within a minute or so. It will be understood that the scene can have different contents: outdoor / indoor, single / multiple objects, etc. The plurality of images may comprise a photo collection or one or more video and may include any number of images, although more images / longer videos are can be advantageous. In some cases the user may 10 be instructed or guided towards providing images that are considered desirable for processing by the method, e.g. ones having diversity in viewing angles / camera poses amongst images, initially static video of the scene, images without moving objects, avoiding lighting changes, etc. 15

[041] At step 206 embodiments can assign a score to each of the obtained plurality of images. The score will typically be in the form of a numerical (or alternative, e.g. grade A - F) value assigned to each image. The score will represent a qualitative and / or quantitative measure of a particular characteristic of, or related to, the image. For example, in some embodiments the score will represent a level of noise (distortion) present in the images and 20 the method may be intended to suggest / create a new image of the scene that has a lower noise level than any of the existing images. In cases such as these the scoring system may be such that a high (desirable) score corresponds to a low noise level.

[042] The scoring of step 206 may use a scoring model. Ideally, the model will assess images 25 independently of image aspect-ratio, image size or image theme (i.e. semantics) and will also offer fine granularity, i.e. ability to assess the characteristic in images that are similar to each other. In some embodiments the model may comprise a machine learning (ML) model, such as a neural network. The ML model can be trained using training data comprising a set of images and a score associated with each of the images. For example, the training data may 30 comprise images that have been scored by one or more human, e.g. who has assigned a score representing a feature of the image such as the level of noise / distortion present. In some embodiments the model can use a MobileNetv2 backbone for feature extraction. This can be followed by two modules, where the first one may comprise an adaptive pooling operation to reduce the spatial size of the feature maps. The second module can comprise 3 linear layers, 35 where ReLU activations are used in between these layers. The final output can comprise a single score. In some embodiments the model may be trained with an ADAM optimizer using 13 08 25 Stochastic Gradient Descent algorithm. This training updates the weights of the network using the training data as the ground-truth.

[043] In some embodiments the score can represent a level of blur present in the images 5 and the method may be intended to suggest / create a new image of the scene having less blurring. In other embodiments the score can represent a quality of the images and the method may be intended to suggest / create an image of the scene having higher quality. Quality may refer to technical considerations, such as resolution, in addition to aesthetic quality. In other embodiments the score can represent an editing score for the images and the method may 10 be intended to suggest / create an image with desired edits. Such editing tasks may require external models (e.g. image inpainting).

[044] In some embodiments the scoring may be based on a model that assigns images scores representing quality based on how well they comply with universal photo-composition 15 rules. Such rules can result in a high score for images based on factors such as one or more of the following: having a plain background, being based the rule of thirds, being based on the rule of odds, the presence of diagonal lines, depth of view, balance of elements, symmetry and / or natural framing, etc. Not only will images having a high score in this regard be more appealing visually but they also have features that can have technical benefits, e.g. improve 20 results for image processes such as reverse image searches, object detection, zooming, improved resolution, etc. Thus, embodiments may be composition-based for generalization and are not limited to a particular user’s aesthetic preferences like existing image suggestion techniques. In some embodiments the scoring model may be based on a modification of the known composition-based SAMP technique, where certain parts of the technique may be 25 changed and / or omitted. For example, embodiments may use a scoring model that may be modified compared to conventional SAMP in all or some of the following ways: saliency prediction is removed; attribute supervision is removed; a single pattern is used instead of several; single score prediction is used instead of predicting N number of scores, and / or training takes place using multiple datasets rather than only one. The inventors have found 30 that this can work significantly faster than conventional SAMP whilst providing comparable, or improved, results.

[045] Alternative methods can use non-ML methods for scoring, e.g. edge detection, background detection, etc, as well as symmetry detection methods and monocular depth cues. 35 In some cases edge detection methods can detect the lines for rule-of-thirds, for example. If detected edges are in line with the rule-of-thirds (which is a preset rule and so what is expected 13 08 25 is known), the method can provide a higher score. Monocular depth cues provide information on the depth of the scene (per-pixel distance to the camera). The range of depth values can be helpful to find whether we have a depth-of-field effect in the image (i.e. little depth variance in an image can indicate no depth-of-field effect is present, resulting in a zero / low score). For 5 background / symmetry detection, embodiments can leverage a saliency method that indicates the salient parts of the image. This can be used for object emphasis rule, as well as the symmetry rule, for example. Saliency methods produce a binary mask, where the salient object is presented in white, while the rest is presented in black. The score in this case may correspond to the size of the white area in the mask. Any of these, or other, outputs can be 10 used to generate a numerical (or other) score, e.g. within a particular numerical range, to an image.

[046] At step 208 the method may perform a check as to whether any of the obtained images have a score that is greater than a threshold. The threshold may be predetermined / prestored 15 by a developer of the application or may be set by a user. For example, if the score assigned at the step 206 can be between 0 and 5, the threshold may be set as 3.35, which should result in an image with a higher than average score. In some cases the threshold may be set as higher than a highest score amongst the scores assigned to the obtained plurality of images. If one or more image has a score greater than the threshold then control may pass to step 210 20 where a high score image is output by the method. If there is more than one image with the same high score then one may be selected, e.g. at random, for output at the step 210. If an image having a score greater than the threshold is not identified at the step 208 then control passes to step 212 so that a new image of the scene may be generated and output. Further, in some cases even if an image having a score greater than the threshold is present amongst 25 the obtained plurality of images then the method may, in any case, offer the user an option to generate a new image of the scene that should have an even better score and pass control to step 212.

[047] At step 212 the method can estimate the camera pose of each of the plurality of 30 obtained images. This information can be used by the method for determining how to render a novel image of the scene. The camera pose information can be in any suitable form, e.g. x / y / z coordinates, rotation matrix, etc. The camera pose may correspond to the position of a device that was used to capture an image, e.g. photograph, or it may correspond to a location of a viewer / observer if the image was artificially generated. The step 212 can be performed 35 using any suitable technique, preferably one that has desirable properties, such as fast 13 08 25 execution time and accurate results. Examples of suitable techniques that can be used to implement the camera pose estimation include COLMAP library, OpenSFM, etc.

[048] At step 214 embodiments can obtain a rendering model 216 usable to render an image 5 relating to the scene, e.g. a new view of the scene from a selected camera pose (viewer / observer position) that will be different to those of the existing images. The rendering model may comprise a ML model, such as a neural network. In typical embodiments the model may be based on a Neural Radiance Field (NeRF) network. This network can take as input a spatial location and viewing angle and can output the volume density and colour at that spatial 10 location. To render an image comprising a view of the scene, rays are shot from each pixel and several sampled points along each ray are evaluated with the NeRF network. The colours along a ray are integrated to produce the final colour for each pixel. NeRF can significantly compress the amount of storage required for rendering. 15

[049] The NeRF model may be initially trained using ground-truth images with known poses, and by comparing the ground-truth image with images rendered by the NeRF. The NeRF network may comprise a number of linear layers. The number can vary, but they are accompanied by ReLLI activations throughout the network, except the final linear layer. In some cases, the network training can use ADAM optimizers with Stochastic Gradient Descent 20 algorithm to update weights of the network. Embodiments may use any suitable NeRF technique, especially one that performs fast, consumes less memory and requires as few input images as possible because this will improve the overall performance / accuracy of the method and provide a better experience. A non-exhaustive list of suitable examples of suitable NeRF techniques includes: NeRF (see, for example, https: / / github.com / bmild / nerf), MobileNeRF 25 (see, for example, https: / / mobile-nerf.github.io / ), Plenoxels (see, for example, https: / / ar5iv.labs.arxiv.org / html / 2112.05131) and MF-NeRF (see, for example, https: / / arxiv.org / pdf / 2304.12587v3.pdf).

[050] Once the model 216 is trained for a scene, embodiments can determine a new camera 30 pose that is predicted to render an image comprising a view of the scene that will have a high score. The selected camera pose can be input into the trained model, which will render the image. The data used for the training can include the plurality of images obtained at the step 204 and the camera poses of each of these images estimated at the step 212. A goal of the training is to learn the representation of the scene with a neural network and the output will 35 comprise a neural network representing the scene. Each different scene will require new training. The training can be performed by the computing device 100. Alternatively, the 13 08 25 computing device can transfer the images and associated camera poses to a server, with the server performing the training. If the training is performed on a server then model weights will be returned to the computing device. In any case, the trained neural network will be relatively small in terms of data size and so can efficiently be stored on, and used by, the computing 5 device and also makes server-client communication efficient, if used.

[051] At step 218 embodiments can seek a new camera pose that is predicted will render a new image of the scene that can have a relatively high score, e.g. a score that may be higher than the best score amongst those of the obtained plurality of images and / or at least equal to 10 a predetermined score threshold. Figure 3 is a flowchart showing example steps involved in this new camera pose seeking. More than one iteration of these steps may be performed by embodiments.

[052] At step 301 embodiments can obtain the scores computed for the plurality of images 15 at the step 206 and the camera poses of images estimated at the step 212. At step 302 embodiments can attempt to recognize a pattern that may exist in relation to the camera poses and their computed scores (i.e. “How do scores change with respect to camera pose components?”). The existence of such a pattern can be used to determine an initial camera pose for the new camera pose seeking process 218. This can be done by leveraging the 20 information that is already available, in particular the computed scores and the estimated camera poses of the images (which were also used fortraining the rendering model 216). The training images will often comprise user-taken images and so will reflect the camera movements of the user, which can help the process recognize the intention of the user. Embodiments can be guided by these movements / trajectory, thereby automatically following 25 the user’s intentions.

[053] In more detail, embodiments can attempt to find a pattern between camera poses of the obtained images (Cn) and their scores (Sn). For example, image In is taken from camera pose Cn, and has an aesthetic score of Sn. The image In can be a user-taken or a computer-30 generated image in some embodiments. This is essentially a pattern recognition problem and so can be solved using any suitable pattern recognition algorithm. One suitable example is clustering, where camera poses and scores are clustered and matching clusters can reveal which camera pose component(s) result in high scores. Neural network-based solutions can also be used. For instance, an embodiment can use neural networks that find the prominent 35 relationships between camera pose components and scores. Such a neural network can take as input N number of camera poses / scores. The network can consist of linear layers and can 13 08 25 use ReLLI activations. The number of layers (i.e. the depth of the network) can be adjusted with respect to hardware requirements. The network can output positive relations between camera pose components and scores, if there are any. The network can be trained with a standard ADAM optimizer, using training data comprising camera poses and related images, 5 and Stochastic Gradient Descent algorithm to update the network weights. Data mining techniques can be used by alternative embodiments, e.g. frequent pattern discovery or K-optimal pattern discovery.

[054] Embodiments can use a recognized pattern / trajectory to intelligently determine the 10 first / initial camera pose to be used by the process 218. A pattern / trajectory is considered to be available if there is a correlation between the scores (S) and camera poses (C) of the images. The correlation can be positive or negative. For example, “Increasing x coordinate of camera pose improves score” is a positive correlation, whilst “Increasing x coordinate reduces score” is a negative correlation. Negative correlations are still meaningful as they show what 15 should not be done during camera pose component adjustment / trajectory exploration. Positive correlations are useful and directly show what should be done during camera pose component adjustment / trajectory exploration in order to find a camera pose that will result in an increased score. Positive correlations are directly useful patterns / trajectories and are more informative in terms of indicating what to do in the next iteration of the pose sampling. On the other hand, 20 negative correlations only cancel out a part of the search space and do not directly show what to do next.

[055] Figures 4A - 4C are diagrams showing examples of correlations between camera pose components and scores. Every point in the plots represents an image and three scenarios are 25 visualised in the Figures 4A - 4C: no correlation available, negative correlation available and positive correlation available, respectively. The examples use X coordinates of camera poses, but it will be understood that other components of the camera pose, e.g. Y or Z coordinates, rotation matrix, etc, can be used. A recognized correlation can be used to determine a camera pose adjustment action that will be performed at the next step (either step 304 or 305 in Figure 30 3).

[056] In the example of Figure 4A, there is no pattern / relation / correlation between X coordinates and the scores. Embodiments may assume that there is also no correlation between other components of camera poses and the scores. This means there is no 35 correlation available and so the action to be performed at the next step (304) will be selecting a pose randomly. 13 08 25

[057] In the example of Figure 4B, there is a negative correlation between X coordinates and the scores, i.e. scores of images tend to decrease as the value of the X coordinate of their camera poses increases. Embodiments may assume that there is no correlation between 5 other components of camera poses and the scores. This pattern indicates that there is a negative correlation and so the action to be performed at the next step (304) will be sampling a pose randomly, excluding increasing the X coordinate.

[058] In the example of Figure 4C, there is a positive correlation between X coordinates and 10 the scores, i.e. scores of images tend to increase as the value of the X coordinate of their camera poses increases. Embodiments may assume that assumed there is no correlation between other components of camera poses and the scores. This pattern indicates there is a positive correlation and so the action to be performed at the next step (305) will be sampling a pose based on this correlation, i.e. by increasing the X coordinate. 15

[059] Returning to Figure 3, if no positive correlation is recognized (e.g. as per the examples of Figures 4A or 4B) at the step 302 then control passes to step 304, where a random initial camera pose is selected (instead of one directly based on a recognized positive correlation). If a negative correlation was found at the step 302 then this random selection may exclude a 20 selection based on that, i.e. the random selection may be rejected and re-executed if it corresponds to the negative correlation. For instance, based the example of Figure 4B, the randomly-selected camera pose will be rejected if there is an increase in its X coordinate compared to the camera pose of the previous iteration. If the current iteration is the first time step 304 is being executed then the camera pose of the image amongst the obtained plurality 25 of training images that had the highest computed score will be used for this comparison. In some embodiments the random selection may be made from among a set of possible actions that can be applied successively. Typical examples of the actions include: increase X coordinate by a value x1; decrease X coordinate by x2; increase Y coordinate by y1; decrease Y coordinate by y2, and so on. Embodiments can select an action based the correlation found. 30 For example, if increasing the value of the X coordinate was found to be a negative correlation then that is excluded from the set of actions for the random selection. This sampling can take one or many action, which itself can be randomized too. The random selection can also be limited by the range of the estimated camera poses of the obtained plurality of images, etc. 35

[060] If a positive correlation is recognized (e.g. as per the example of Figure 4C) at the step 302 then control passes to step 305. At the step 305 a camera pose is selected using an 13 08 25 adjustment action based on the positive correlation. For example, if the positive correlation indicates that the X coordinate should be increased (e.g. as per the example of Figures 4C) then the X coordinate of the camera pose of the previous iteration is increased to produce the selected camera pose. If the current iteration is the first time step 305 is being executed then 5 X coordinate of the camera pose of the image amongst the obtained plurality of training images that had the highest computed score will be increased.

[061] Following the step 304 or 305, control passes to step 306, where the rendering model 216 is used to render an image based on input comprising the selected camera pose. Here, 10 the resolution of the rendered image will be lower than the resolution of the obtained plurality of images, e.g. up to 4 - 8 times lower to save time and resources. To give an example, the resolution of the obtained images may be 3088x1440 (i.e. Samsung S23 Ultra display resolution), while the low-resolution may be 772x360 (4 times smaller), although it will be understood that many variations are possible. 15

[062] At step 308 embodiments can compute a score for the lower resolution image rendered at the step 306. The score will normally be computed using the same scoring model as used at the step 206. 20

[063] At step 310 embodiments can determine whether the image score computed at the step 308 is greater than a predetermined threshold. The threshold may be predetermined by a developer or may be set by a user. For example, if the score assigned at the step 308 can be between 0 and 5 then the threshold may be set as 3.35, which should provide an image with a sufficiently higher than average score. In some cases the threshold may be set as higher 25 than a highest score amongst the scores assigned to the obtained plurality of images. If the computed score is greater than the threshold then the image that will be rendered by the corresponding camera pose is considered to be sufficient and so control passes to step 220, where the currently selected camera pose (i.e. the one used to generate the most recent low resolution image at the step 306) is output. If the computed score is not greater than the 30 threshold then control passes to step 312.

[064] At step 312 embodiments can determine whether the current iteration of the process 218 exceeds an iteration threshold (e.g. first iteration = 1 and the iteration threshold is 10). The threshold may be predetermined by a developer or may be set by a user, e.g. based on 35 a desired execution time / duration and / or output image quality. If the iteration threshold has 13 08 25 been reached then control passes to step 220, where the currently selected camera pose is output. The score of the resulting image will still normally exceed the highest score amongst the existing plurality of images. If the iteration threshold has not been reached then control passes to step 314. 5

[065] At step 314 embodiments can seek to determine a new camera pose that has a high probability of rendering an image having a score higher than the score of the image rendered using the currently selected camera pose. Figure 5 is a flowchart showing example steps involved in this process. 10

[066] At step 502, the score (“current score”) of the image rendered using the currently selected camera pose and the score (“previous score”) of the image rendered using the camera pose selected at the previous iteration of the process 218 are obtained. If the current iteration is the initial / first time step 502 is being executed then the previous score will comprise 15 the image amongst the obtained plurality of training images that had the highest computed score.

[067] At step 504, the obtained current score and the previous score are compared. The result of this comparison will be used to determine the camera pose adjustment action that 20 will be taken at the next iteration of the step 305 (which follows the step 314). Examples of camera pose adjustment actions include increasing or decreasing the value of the component by 1 or some other number. The different component / adjustment action may be selected at random, e.g. from a set of possible permitted actions (in a similar manner to the step 304 described above), or may be selected based on selection history, etc. In some embodiments 25 the comparison may comprise finding the difference (“gradient”) between the current score and the previous score. In alternative embodiments more complex formulations may be used instead of finding the difference. For example, embodiments may store “history” type information of what happened at each iteration of the exploration to take into account the bigger picture. This can provide a memory bank that allows embodiments to “tolerate” moving 30 in a direction / selecting an adjustment action that failed in the (one or more) immediately previous iteration, but was mostly positive in a longer consideration of previous iterations, e.g. over the last 15 iterations.

[068] If the result of the comparison of the step 504 indicates the current score is greater 35 than the previous score (a “positive gradient”) then the camera pose adjustment action (“trajectory”) that resulted in the current score / camera pose will be continued (step 506B) at 13 08 25 the next iteration of the step 305. That is, the same component of the camera pose will be adjusted in the same adjustment manner at the next iteration of the step 305. For instance, if the current camera pose was selected based on positive correlation / action of “increasing X coordinate increases score” then the X coordinate of the camera pose will be increased again 5 at the next execution of the step 305.

[069] If the result of the comparison of the step 502 indicates the current score is not greater than (or merely equal to) the previous score then a different camera pose adjustment action will be used (step 506A) at the next iteration of the step 305. That is, a different component of 10 the camera pose will be adjusted (in the same adjustment manner) at the next iteration of the step 305, or the same component will be adjusted in a different adjustment manner. For instance, if the current camera pose was selected based on the correlation / action of “increasing X coordinate increases score” then at the next execution of the step 305 the X coordinate may be decreased, or a component other than the X coordinate, e.g. the Y or Z 15 coordinate, may be increased.

[070] Process 314 is quite similar to the pattern recognition problem of process 302, although here the gradient is the change in scores between images rendered using the current and previous camera poses. In each iteration, embodiments can check the gradient and take one 20 of two actions. If the gradient improves the score (positive gradient) then continue with that trajectory. If gradient does not improve the score (negative gradient) then continue using a random trajectory that is not the same as the previous trajectory. The approach can efficiently find the camera pose that can significantly increase or maximize the score. Embodiments are likely to surpass the score threshold within a given iteration limit and are also more likely to 25 output an image having the “best possible” score if / when the iteration limit is reached. Further, the low-resolution images rendered at step 306 do not need to be saved long-term and only the camera pose of the final iteration is output.

[071] Figure 6 includes a first graph 602 that illustrates computed scores of the images 30 rendered by iterations of the step 306 of the new camera pose seeking process 218 along its x-axis, with the y-axis representing the iterations. As can be seen, the scores generally improve over time, meaning that the score of the image rendered using the camera pose selected at the final iteration has a high probability of being a high score, e.g. that exceeds the threshold. 13 08 25

[072] To contrast, graph 604 illustrates computed scores of images generated using a naive, brute-force approach. This approach randomly samples a pose in the scene and renders a full-resolution image using that pose. The score of the rendered image is computed and the the rendered image and its score are saved in a database. This process is repeated until an 5 iteration limit is reached, or an image is rendered with a score above a threshold. However, there are problems with this approach. In particular, it is costly to render a full-resolution image each time. There is also no guarantee that the threshold will be reached. The complete randomness of the process means that performance is unstable and it does not exploit already-computed scores and so is inefficient. Further, there is a need to save all intermediate 10 images, which requires additional storage.

[073] In comparison, embodiments can leverage the computed scores of the training images and so are more efficient, and also render temporary images during processing in low-resolution to save time and computation. Embodiments can also “follow” the clues in pose 15 sampling and so are stable and are also practically guaranteed to produce the best image in terms of scoring. Embodiments also avoid the need to save intermediate images and so have low storage requirements.

[074] Returning to Figure 2, after the selected camera pose is output at step 220, it is used 20 to generate an image at step 222. This will typically be done using the trained rendering model 216. The resulting image 224 can be displayed to the user on the computing device 100 and / or stored and / or transferred to another device. The rendered image may be used / processed in any suitable manner, e.g. to provide a suggestion to the user that they should share their best scored images on social media; guide the user while taking photos, thereby reducing the 25 photography expertise required to produce good quality images; use the image for an imagebased search or object recognition process, etc.

[075] Alternative embodiments can provide an intelligent photography assistant that guides the user to improve photos in real-time. Embodiments can also be used to power other use 30 cases, such as image animation and lifting video / image editing capabilities to 3D for further fidelity. The optimization-based approach for scene exploration can be implemented with different algorithms (not just the new camera pose seeking process described herein), which can help tailor performance for different hardware. Alternative embodiments may use more complex exploration, such as using a completely separate NN to determine how to produce 35 the trajectory instead of following the gradient. LO CXI O CO

[076] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will 5 recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims. 13 08 25

Claims

1. A computer-implemented image processing method comprising:5 obtaining a plurality of images, each of the plurality of images comprising a view of ascene;computing a respective score for each of the plurality of images;10 estimating a respective camera pose of each of the plurality of images;using the computed scores and the estimated camera poses to determine a new camera pose useable for generating a new image comprising a view of the scene and having a score greater than a threshold, and15generating the new image using the new camera pose,wherein the computed scores are computed using a scoring model comprising a trained machine learning model, and the computed scores represent a characteristic of an 20 image, andwherein the computed scores represent a characteristic comprising a level of noise of an image, a level of blurring of the image, or a quality of the image based on an extent to which the image complies with universal photograph composition rules.

252. A method according to claim 1, further comprising:applying a pattern recognition process to the estimated camera poses and the computed scores to find a camera pose amongst the estimated camera poses associated with30 a high score amongst the computed scores, andselecting the camera pose found to have the high score as an initial camera pose for use in the step of determining the new camera pose, or35 if a camera score having a high score is not found then selecting a random camerapose as the initial camera pose for use in the step of determining the new camera pose.13 08 253. A method according to claim 2, further comprising:rendering an image using the initial camera pose, the rendered image having a 5 resolution lower than a resolution of the obtained plurality of images;computing a score for the low resolution image using the scoring model;determining whether the computed score of the low resolution image is greater than a 10 predetermined threshold, andif the computed score of the low resolution image is greater than the predetermined threshold then outputting the camera pose used to generate the low resolution image as the new camera pose, or15if the computed score of the low resolution image is not greater than the predetermined threshold then seeking a candidate new camera pose.

4. A method according to claim 3, wherein the step of seeking the candidate new camera 20 pose comprises adjusting a component of the candidate new camera pose based on whether an image generated using the candidate new camera pose having the component adjusted improves or deteriorates the score of the generated image compared to a score of an image generated using a camera pose not having the component adjusted.25 5. A method according to claim 3, wherein the step of seeking the candidate new camerapose comprises:adjusting a component of the candidate new camera pose at a current iteration of the method in a particular adjustment manner to produce a modified camera pose;30generating an image based on the modified camera pose, the image generated based on the modified camera pose having a lower resolution than a resolution of the obtained plurality of images;35computing a score for the image generated based on the modified camera pose;13 08 25determining whether the computed score is greater than a previous score computed for an image generated using a previous camera pose during a previous iteration of the method;5 if the computed score is greater than the previous score then adjusting the componentof the camera pose in the particular adjustment manner at a next iteration of the method, andif the computed score is not greater than the previous score then adjusting a different component of the camera pose in the particular adjustment manner at the next iteration, or10 adjusting the component of the camera pose in a different adjustment manner at the next iteration.

6. A method according to claim 5, wherein:15 the component of the camera pose comprises an x-coordinate, y-coordinate or z-coordinate of the camera pose, or a rotation matrix of the camera pose, andthe adjustment manner comprises increasing or decreasing a value of the coordinate, or making a clockwise or counter-clockwise rotation of the rotation matrix.

207. A method according to claim 5 or 6, wherein the adjustment manner is selected from a set, and wherein a said adjustment manner is excluded for selection from the set if the computed score is not greater than the previous score.25 8. A method according to any of claims 5 to 7, including using a pattern recognitionprocess to identify an initial adjustment manner to be used in the step of seeking the candidate new camera pose, and wherein the initial adjustment manner to be used in the step of seeking the candidate new camera pose is randomly selected if a camera score having a high score is not identified.

309. A method according to any of claims 5 to 8, further comprising:comparing the current iteration to an iteration threshold, and35 if the current iteration is greater than the iteration threshold then outputting the modifiedcamera pose as the new camera pose.13 08 2510. A method according to any preceding claim, wherein the step of generating the new image using the new camera pose is performed using a machine learning model, wherein the machine learning model comprises a Neural Radiance Field, NeRF, model trained using 5 training data comprising camera poses and images to output an image rendered based on an input camera pose.

11. A method according to any preceding claim, further comprising:10 determining whether at least one of the computed scores is greater than a scorethreshold, andif at least one of the computed scores is greater than the score threshold then outputting the image of the plurality of images having the score greater than the score 15 threshold, andif at least one of the computed scores is not greater than the score threshold then performing the step of estimating the respective camera pose of each of the plurality of images.2012. A computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry a method according to claim 1.

13. Apparatus adapted to perform image processing, the apparatus comprising at least 25 one processor configured to execute a method according to claim 1.

Images