Image processing method, and associated image processing device and display system

Abstract

The invention relates to a method (20) for processing at least one image, the method (20) being implemented by computer and comprising, for each current image (4), the following steps: - with the current image (4) being displayed on at least one first display medium (6), detecting (22), in the displayed image (4) and for each observer (10) among a plurality of observers (10), an area of interest (8) of the observer (10); - computing (24) a general area of interest (32) from each detected area of interest (8); and - applying (26) a differentiated image processing operation to the displayed image inside and outside the general area of interest (32), in order to obtain a processed image (16).

Description

Image processing method, image processing device and display system associated therewith DESCRIPTION Technical field

[0001] The present invention relates to a method for processing at least one image.

[0002] The invention also relates to a computer program and a device implementing such a method.

[0003] The invention applies to the field of displaying a digital image on a display medium, such as a display screen or a projection surface. State of the art

[0004] It is known to use graphical elements rather than textual ones to convey information quickly and concisely. Such graphical elements include, for example, graphs, icons, symbols, animations, etc.

[0005] Such elements are, for example, implemented in the context of conferences or meetings during which a speaker seeks to share information with a plurality of participants.

[0006] However, this method of communication is not entirely satisfactory.

[0007] Indeed, it is generally not desirable to allow observers to react in real time to the information presented by the speaker, in order to avoid a cacophony generated by a multitude of simultaneous questions and / or comments.

[0008] As a result, the speaker is unable to determine how the audience perceives the graphic elements used to support the information he is trying to convey.

[0009] Therefore, the speaker is unable to ensure that the observers' attention is directed to the graphic element that he considers relevant at the current moment, or is likely to fail to realize that there is an unclear point in his presentation on which the observers' attention is focused.

[0010] An aim of the present invention is to remedy at least one of the drawbacks of the state of the art.

[0011] Another aim of the invention is to propose a system allowing a speaker to determine the element(s) on which the attention of observers is focused. Statement of the invention

[0012] To this end, the invention relates to an image processing method of the aforementioned type, the method being implemented by computer and comprising, for each current image, the following steps: - the current image being displayed on at least a first display medium, detection in the displayed image, for each observer among a plurality of observers, of an area of ​​interest of said observer; - calculation of a global area of ​​interest from each detected area of ​​interest; and - application, to the displayed image, of differentiated image processing in and outside the global area of ​​interest, to obtain a processed image.

[0013] Indeed, the calculation of the global area of ​​interest provides access to information representative of the areas of interest detected for all observers, that is to say of each part of the displayed image on which an observer's attention is focused.

[0014] As a result, thanks to differentiated image processing, the processed image is likely to highlight one or the other of the overall area of ​​interest or its complement in the image, thus giving the speaker information on the elements towards which the attention of the observers is directed, for example in real time.

[0015] Advantageously, the method according to the invention has one or more of the following characteristics, taken in isolation or in any technically possible combination:

[0016] for each observer, the corresponding area of ​​interest is a part of the displayed image which is observed by said observer;

[0017] Image processing includes: - a modification of at least one characteristic of at least one pixel, preferably a hue, saturation and / or brightness of a color of each pixel; and / or - a change in sharpness;

[0018] the calculated global area of ​​interest depends on each area of ​​interest detected during at least one detection instant belonging to a predetermined time window;

[0019] the predetermined time window extends between an initial instant of display of the current image and a current instant;

[0020] the calculated overall area of ​​interest further depends on a weighting function of the detected areas of interest, the weighting function having a predetermined evolution over the predetermined time window;

[0021] the weighting function is a decreasing function of seniority;

[0022] the calculated global area of ​​interest depends on each area of ​​interest that is detected during a predetermined inclusion time interval, and which is associated with a part of the current image for which no area of ​​interest has been detected during a predetermined exclusion time interval, distinct from the predetermined inclusion time interval;

[0023] the global area of ​​interest associated with the current image is, furthermore, a function of the global area of ​​interest associated with a previous image, if a similarity calculated between the current image and the previous image is greater than a predetermined threshold;

[0024] the method comprises, during an update of the current image, a resetting of the global area of ​​interest if a similarity calculated between the current image and the previous image is less than a predetermined threshold;

[0025] the method further comprises displaying the processed image on a second display medium;

[0026] the second display medium is distinct from each first display medium.

[0027] According to another aspect of the invention, there is provided a computer program comprising executable instructions which, when executed by a computer, implement the steps of the method as defined above.

[0028] The computer program can be in any computer language, such as machine language, C, C++, JAVA, Python, etc.

[0029] According to another aspect of the invention, a device is proposed for processing at least one image, configured so as to, for each current image displayed on at least one first display medium: - detecting, in the displayed image, for each observer among a plurality of observers, an area of ​​interest of said observer; - calculate a global area of ​​interest from each detected area of ​​interest; and - apply, to the displayed image, differentiated image processing inside and outside the global area of ​​interest, to obtain a processed image.

[0030] The device according to the invention may be any type of device such as a server, a computer, a tablet, a calculator, a processor, a computer chip, programmed to implement the method according to the invention, for example by executing the computer program according to the invention.

[0031] According to another aspect of the invention, there is provided a display system comprising a processing device as defined above and at least one second display medium configured to display each processed image obtained.

[0032] Preferably, the display system is a smart mobile phone (or "smartphone" in English), also called a ordiphone or multifunction mobile phone).

[0033] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the smart mobile telephone, to obtain a processed image from at least one current image displayed on at least one first display medium.

[0034] Alternatively, the display system is a touchscreen tablet.

[0035] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the touch tablet, to obtain a processed image from at least one current image displayed on at least one first display medium.

[0036] Alternatively, the display system is a computer.

[0037] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the computer, to obtain a processed image from at least one current image displayed on at least one first display medium.

[0038] Alternatively, the display system is a television or a display terminal.

[0039] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the television or the display terminal, to obtain a processed image from at least one current image displayed on at least one first display medium.

[0040] Alternatively, the display system is a virtual reality headset or an augmented reality headset.

[0041] In this case, the invention also relates to a use of a processing device according to the invention and / or of a method according to the invention, within the virtual reality headset, or within the augmented reality headset, to obtain a processed image from at least one current image displayed on at least one first display medium.

[0042] Alternatively, the display system is included in a medical imaging device, for example an endoscope.

[0043] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the medical imaging apparatus, to obtain a processed image from at least one current image displayed on at least one first display medium.

[0044] Alternatively, the display system is included in a vehicle, autonomous or not, such as a land vehicle (e.g., a car), an aircraft (e.g., a drone, an airplane, a helicopter, etc.) or a maritime vehicle (e.g., a boat, a submarine, etc.).

[0045] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the vehicle, to obtain a processed image from at least one current image displayed on at least one first display medium. Brief figures

[0046] The invention will be better understood upon reading the following description, given solely as a non-limiting example and with reference to the appended drawings in which:

[0047] Figure 1 is a schematic representation of a display system according to the invention; and

[0048] Figure 2 is a flowchart of an image processing method implemented by the display system of Figure 1.

[0049] It is understood that the embodiments which will be described below are in no way limiting. In particular, it will be possible to imagine variants of the invention comprising only a selection of characteristics described below isolated from the other characteristics described, if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection includes at least one preferably functional characteristic without structural details, or with only a part of the structural details if it is this part which is only sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.

[0050] In particular, all the variants and embodiments described can be combined with each other if there is no technical obstacle to this combination.

[0051] In the figures and in the rest of the description, the elements common to several figures retain the same reference. Detailed description

[0052] A display system 2 according to the invention is illustrated in Figure 1.

[0053] Preferably, the display system 2 is a smart mobile phone, a touch pad, a computer, a television, a display terminal, a virtual reality headset or an augmented reality headset. Alternatively, the display system is included in an imaging device medical, for example an endoscope, or included in a vehicle, autonomous or not, such as a land vehicle (for example, a car), an aircraft (for example, a drone, an airplane, a helicopter, etc.) or a maritime vehicle (for example, a boat, a submarine, etc.).

[0054] The display system 2 is, in particular, intended to determine, in an image 4 displayed on at least a first display medium 6, each zone 8, called “zone of interest” on which the attention of each of a plurality of observers 10 is focused.

[0055] Preferably, the display system 2 is also intended to carry out such a determination for each of a plurality of images displayed successively on the first display support(s) 6, such as successive images of a video stream.

[0056] The display system 2 is associated with at least one first display support 6. Alternatively, each first display support 6 (also called “first support”) is included in the display system 2.

[0057] For example, each first display medium 6 is a display screen, such as a screen of a smartphone, a tablet, a computer, a television, a display terminal, a virtual reality and / or augmented reality headset, etc.

[0058] According to another example, each first display medium 6 is a surface onto which an image is projected, such as for example a display surface associated with a projector, etc.

[0059] According to another example, each first display support 6 is a display support of a device worn by a corresponding observer 10, for example a pair of glasses, or glasses, a virtual reality headset, an augmented reality headset, etc.

[0060] The display system 2 comprises a processing device 12 and at least one second display medium 14 connected to the processing device 12.

[0061] For example, each second display medium 14 is a display screen, such as a screen of a smartphone, a tablet, a computer, a television, a display terminal, a reality headset virtual and / or augmented reality, a dashboard, a medical imaging device, etc.

[0062] According to another example, each second display medium 14 is a surface onto which an image is projected, such as for example a display surface associated with a projector, etc.

[0063] According to another example, each second display support 14 is a display support of a device worn by a corresponding person (such as an observer 10 or a third person), for example a pair of glasses, or glasses, a virtual reality headset, an augmented reality headset, etc.

[0064] For example, each second display support 14 is merged with a first display support 6. Alternatively, each second display support 14 is distinct from each first display support 6.

[0065] The processing device 12 is configured to determine, in each displayed image (called “current image”), each area of ​​interest 8 on which the attention of an observer 10 is focused, and to generate, as output, at least one processed image 16 associated with the current image 4 and representative of the result of such a determination.

[0066] Further, each second display medium 14 (also referred to as a "second medium") is configured to display each processed image 16 generated by the processing device 12.

[0067] To make such a determination, the processing device 12 is configured to implement an image processing method 20, illustrated by FIG. 2.

[0068] As appears in this figure, the image processing method 20 comprises a step 22 of detecting areas of interest (called “detection step”), a step 24 of calculating the overall area of ​​interest (called “calculation step”) and a step 26 of image processing (called “processing step”).

[0069] Optionally, the image processing method 20 includes a step 28 of displaying the processed image (called a “display step”). Detection step 22

[0070] More precisely, the processing device 12 is configured to, during the detection step 22, detect, in the current image 4 displayed on each first support 6, an area of ​​interest associated with each observer 10 among the plurality of observers 10.

[0071] Advantageously, each detected area of ​​interest 8 is associated with the corresponding detection time. Such a feature is advantageous insofar as, as will be described later, it allows a selection of the detected areas of interest 8 on the basis of their respective detection times.

[0072] Preferably, for each observer 10, the corresponding area of ​​interest is the part of the current image 4 which is observed by said observer 10. Alternatively, or in a complementary manner, for each observer 10, the corresponding area of ​​interest is the part of the current image 4 which is pointed at by said observer 10, by means of a part of his body (such as a finger) or an object (such as a stylus).

[0073] In this case, the processing device 12 is preferably associated with an imaging system 30 for imaging the observers 10. Furthermore, in this case, the processing device 12 is configured to determine, from an output of the imaging system 30, the area 8 of interest associated with each observer 10 as being the part of the current image 4 which is observed and / or pointed at by said observer 10.

[0074] For example, for each observer 10, the corresponding area of ​​interest 8 is defined as an area around a point of intersection of a display surface of the first display medium 6 with a direction of gaze of said observer 10 and / or with a pointing direction of said observer 10.

[0075] In the example of Figure 1, the lines Di, D2 and D3 illustrate the directions of gaze of three observers 10, respectively.

[0076] Furthermore, the processing device 12 is configured to calculate, during the calculation step 24, a global area of ​​interest 32 from each detected area of ​​interest 8.

[0077] For example, the processing device 12 is configured to calculate the overall area of ​​interest 32 as the result of applying a logical function to all of the detected areas of interest 8.

[0078] Preferably, in this case, at a given instant, the global area of ​​interest is the result of the union of all the areas of interest 8 detected for said instant. In this way, the global area of ​​interest 32 is indicative of all the parts of the current image 4 on which the interest of the observers 10 is focused. In the example of FIG. 1, the global area of ​​interest 32 is defined as the union of three areas of interest 8 detected.

[0079] Alternatively, at a given instant, the global area of ​​interest 32 is the result of the intersection of at least two areas of interest 8 detected for said instant. In this way, the global area of ​​interest 32 is representative of each part of the current image 4 on which the interest of several observers 10 is simultaneously focused.

[0080] In the case where the display of the current image 4 persists over time, the processing device 12 is advantageously configured to calculate the global area of ​​interest 32 as a function of each area of ​​interest 8 detected during at least one detection instant belonging to a predetermined time window, called the “observation window”. As a result, the global area of ​​interest 32 is likely to depend on the evolution, during the observation window, of the area of ​​interest 8 associated with each observer 10.

[0081] For example, the observation window extends between an initial instant of display of the current image 4 and a current instant. In this case, at the current instant, the global area of ​​interest 32 is a function of the areas of interest 8 detected since the display of the current image 4 on the or each first support 6.

[0082] By "initial instant of display of the current image", is meant, for the purposes of the present application, the instant from which the current image 4 was displayed on the or each first support 6.

[0083] In another example, the observation window is a fixed window delimited by two predetermined dates, or a sliding window delimited by two dates whose time difference from the current time is predetermined.

[0084] Advantageously, the processing device 12 is also configured to calculate the overall area of ​​interest 32 from a weighting function of the areas of interest 8 detected according to their detection time. In this case, the weighting function has a predetermined evolution during the observation window.

[0085] Such a feature is advantageous, insofar as it allows the influence, in the calculation of the global area of ​​interest, of each detected area of ​​interest to be modulated according to the respective detection date.

[0086] Preferably, the weighting function is a decreasing function of the age of the detected areas of interest 8. In this case, the older a detected area of ​​interest 8 is (i.e. the further its detection time is from the current time), the lower its influence in the calculation of the overall area of ​​interest 32.

[0087] Advantageously, the processing device 12 is configured to calculate the overall area of ​​interest 32 as a function of each area of ​​interest 8: - which is detected during a predetermined inclusion time interval; and - which is associated with a part of the current image 4 for which no area of ​​interest has been detected during a predetermined exclusion time interval, distinct from the predetermined inclusion time interval.

[0088] Such a characteristic is advantageous, insofar as it allows the highlighting of the parts of the current image 4 on which the attention of the observers 10 is drawn during a time interval of inclusion relevant to a given situation, but which were not the subject of interest of the observers during a time interval of exclusion not relevant to said situation.

[0089] For example, such a feature is likely to highlight parts of the current image 4 that have been of recent interest without having been of previous interest. In this way, newly observed parts of the current image 4 are highlighted.

[0090] Such a feature is also likely to highlight parts of the current image 4 that observers 10 are no longer interested in, even though they were previously of interest to them. In this way, parts of the current image 4 whose attention has recently been abandoned are highlighted.

[0091] Preferably, in the case of displaying a new image on the first medium 6 (for example, the next image of a video stream), the processing device 12 is configured to calculate a similarity between the newly displayed current image 4 and the previous image (i.e. the previously displayed image).

[0092] In this case, the processing device 12 is advantageously configured to calculate the global area of ​​interest 32 associated with the current image 4 as a function of the global area of ​​interest associated with the previous image, if the calculated similarity is greater than a first predetermined threshold.

[0093] This characteristic is advantageous, insofar as it leads to retaining a memory of the attention of the observers 10. This is all the more relevant in the case where the successively displayed images are images from a video stream representative of a scene which changes little.

[0094] Alternatively, or additionally, the processing device 12 is advantageously configured to reset the global area of ​​interest 32 if the calculated similarity is less than a second predetermined threshold. second predetermined threshold is, for example, equal to the first predetermined threshold.

[0095] This feature is advantageous, insofar as the parts of the new current image 4 on which the interest of the observers 10 is focused are potentially decorrelated from the parts which were of interest in a previous image different from the current image 4. Processing step 26

[0096] Furthermore, the processing device 12 is configured to apply, during the processing step 26, a processing to the current image 4, in order to obtain a processed image 16. More precisely, to obtain the processed image 16, the processing device 12 is configured to apply, to the current image 4, a differentiated image processing in and outside the calculated global area of ​​interest 32.

[0097] In other words, the processing device 12 is configured to apply, to the part of the displayed image corresponding to the overall area of ​​interest 32, a processing distinct from a processing applied to the complementary part of the displayed image 4 (i.e. which does not correspond to the overall area of ​​interest 32).

[0098] Such differentiated image processing is advantageous, insofar as it allows the highlighting of one or other of the part of the current image associated with the global area of ​​interest 32 and its complement in the current image 4.

[0099] For example, to implement such image processing, the processing device 12 is configured to modify at least one characteristic of at least one pixel, preferably a hue, a saturation and / or a brightness of a color of each pixel. For example, the processing device 12 is configured to increase the brightness of the pixels of the part of the current image 4 corresponding to the overall area of ​​interest 32.

[0100] Alternatively, or in a complementary manner, to implement such image processing, the processing device 12 is, for example, configured to modify the sharpness of the current image 4. For example, the processing device 12 is configured to reduce the brightness of the pixels of, and / or the sharpness of, the portion of the current image 4 which does not correspond to the overall area of ​​interest 32.

[0101] In the example of Figure 1, the part of the image corresponding to the overall area of ​​interest 32 is left sharp, while blurring is applied to the complementary part, which appears grayed out in the figure. Display step 28

[0102] Furthermore, for each current image 14, the processing device 12 is configured to transmit the corresponding processed image 16 to each second medium 14 for its display.

[0103] Advantageously, the second support 14 is distinct from each first support 6. Such a characteristic is advantageous insofar as the second support 14 is likely to be consulted by a third person 34, distinct from the observers 10 and called a “speaker”, wishing to know the parts of the current image 4 on which the attention of the observers 10 is focused. Functioning

[0104] The operation of the display system 2 will now be described with reference to Figures 1 and 2.

[0105] During the detection step 22, the processing device 12 detects, in the current image 4 displayed on each first support 6, the area of ​​interest 8 associated with each observer 10 among the plurality of observers 10.

[0106] Then, during the calculation step 24, the processing device 12 calculates the overall area of ​​interest 32 from each detected area of ​​interest 8.

[0107] Then, during the processing step 26, the processing device 12 applies, to the current image 4, a differentiated processing, in and outside the calculated global area of ​​interest 32, to obtain the processed image 16.

[0108] Optionally, for each current image 14, the processing device 12 then transmits the corresponding processed image 16 to each second medium 14 for display. Examples

[0109] Examples of use of the display system 2 will now be described.

[0110] According to a first example, the second support 14 is merged with the or each first support 6.

[0111] Such a configuration is, for example, envisaged in a situation where it is desirable for observers to be aware of the parts of the displayed image on which the attention of other observers is or has been focused.

[0112] This is, for example, the case in a game whose aim is to discover areas of interest in a displayed image (or a displayed video stream), for example areas representative of a danger, such as avatars of enemy characters.

[0113] In this case, the processing device 12 is preferably configured so that the already identified areas of interest are blurred or darkened, so that the observers focus on other parts of the displayed image which need to be explored.

[0114] According to another example, in the context of a transmission of information (school setting, for example), the second support 14 is distinct from each first support 6, and is intended to be consulted only by the speaker.

[0115] In this way, the speaker is likely to adapt his speech according to the real interest of the observers. In addition, the fact that the second support 14 is distinct from each first support 6 avoids presenting, to the observers, a processed image which would be likely to distract them, or to hinder a good transmission of information.

[0116] Another example concerns the display of images of an advertising and commercial nature. In this case, the processed image is likely to highlight, in a synthetic manner, the product(s) that are of interest to observers (who are, for example, potential customers), from a wider set of products presented.

[0117] Another example concerns the field of medicine, and particularly surgery. For example, during a surgical operation, the observers are several expert doctors, who coordinate the action of a surgeon performing the operation by jointly focusing on specific parts of successive images of a video stream of the operating field.

[0118] Another example concerns the field of transport, for example land or air. In this case, a processed image is displayed to a driver (respectively a pilot), said processed image resulting from the combined observations of other drivers (respectively other pilots) in order to inform him of the presence of dangers or points of interest in a given geographical area.

[0119] Of course, the invention is not limited to the examples which have just been described.

Claims

CLAIMS 1. Method (20) for processing at least one image, the method (20) being implemented by computer and comprising, for each current image (4), the following steps: - the current image (4) being displayed on at least a first display medium (6), detection (22) in the displayed image (4), for each observer (10) among a plurality of observers (10), of an area of ​​interest (8) of said observer (10); - calculation (24) of a global area of ​​interest (32) from each area of ​​interest (8) detected; and - application (26), to the displayed image, of differentiated image processing in and outside the global area of ​​interest (32), to obtain a processed image (16).

2. Method according to claim 1, wherein, for each observer (10), the corresponding area of ​​interest (8) is a part of the displayed image which is observed by said observer (10).

3. Method according to claim 1 or 2, wherein the image processing comprises: - a modification of at least one characteristic of at least one pixel, preferably a hue, saturation and / or brightness of a color of each pixel; and / or - a change in sharpness.

4. Method according to any one of claims 1 to 3, in which the calculated global area of ​​interest (32) depends on each area of ​​interest (8) detected during at least one detection instant belonging to a predetermined time window.

5. Method according to claim 4, in which the predetermined time window extends between an initial instant of display of the current image and a current instant.

6. Method according to claim 4 or 5, in which the calculated global area of ​​interest (32) further depends on a weighting function of the detected areas of interest (8), the weighting function having a predetermined evolution during the predetermined time window.

7. The method of claim 6, wherein the weighting function is a decreasing function of seniority.

8. Method according to any one of claims 1 to 7, in which the calculated global area of ​​interest (32) depends on each area of ​​interest (8) which is detected during a predetermined inclusion time interval, and which is associated with a part of the current image for which no area of ​​interest (8) has been detected during a predetermined exclusion time interval, distinct from the predetermined inclusion time interval.

9. Method according to any one of claims 1 to 8, in which the global area of ​​interest (32) associated with the current image is, in addition, a function of the global area of ​​interest (32) associated with a previous image, if a similarity calculated between the current image and the previous image is greater than a predetermined threshold.

10. Method according to any one of claims 1 to 9, comprising, during an update of the current image, a resetting of the global area of ​​interest (32) if a similarity calculated between the current image and the previous image is less than a predetermined threshold.

11. A method according to any one of claims 1 to 10, further comprising a display (28) of the processed image on a second display medium (14).

12. Method according to claim 11, wherein the second display medium (14) is distinct from each first display medium (6).

13. A computer program comprising executable instructions which, when executed by a computer, implement the steps of the method according to any one of claims 1 to 12.

14. Device (12) for processing at least one image, configured so as to, for each current image (4) displayed on at least one first display medium (6): - detecting, in the displayed image (4), for each observer (10) among a plurality of observers (10), an area of ​​interest (8) of said observer (10); - calculating a global area of ​​interest (32) from each detected area of ​​interest (8); and - applying, to the displayed image (4), differentiated image processing in and outside the global area of ​​interest, to obtain a processed image.

15. Display system (2) comprising a processing device (12) according to claim 14 and at least one second display medium (14) configured to display each processed image (16) obtained.

Images