Image processing apparatus

Abstract

The invention provides an image processing apparatus. The image processing apparatus includes an image acquisition unit, an event acquisition unit, a detection unit, and a determination unit. The image acquisition unit is configured to acquire an image of a scene. The event acquisition unit is configured to acquire an event of the scene. The detection unit is configured to detect edges in a plurality of directions from at least a partial region of the acquired image. The determination unit is configured to determine a direction in which the event acquisition unit moves with respect to the scene according to a detection result of the detection unit.

Description

Technical Field

[0001] The present invention relates to an image processing device, and more particularly to an image processing device capable of appropriately acquiring events. Background Art

[0002] In an image processing device having an event acquisition unit (for example, an event camera), when a scene changes, the event acquisition unit acquires the scene change as an event.

[0003] In an image processing apparatus, predetermined processing such as object recognition and deblurring is performed using events acquired by an event acquisition unit. In this case, if events are not acquired properly and the amount of acquired events is small, the accuracy of the predetermined processing using the events may decrease. Summary of the Invention

[0004] The present invention has been made in view of the above-mentioned problem, and an object of the present invention is to provide an image processing device capable of appropriately acquiring events.

[0005] In order to solve the above problems and achieve the purpose of the present invention, according to one aspect of the present invention, an image processing device is provided, which includes an image acquisition unit, an event acquisition unit, a detection unit, and a determination unit. The image acquisition unit is configured to acquire an image of a scene. The event acquisition unit is configured to acquire an event of the scene. The detection unit is configured to detect edges in multiple directions from at least a portion of the acquired image. The determination unit is configured to determine the direction in which the event acquisition unit moves relative to the scene based on the detection result of the detection unit.

[0006] The beneficial effect of the present invention is that: according to one aspect of the present invention, an image processing device capable of appropriately acquiring events can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram of the external configuration of the image processing apparatus according to the present embodiment.

[0008] Figure 2 is a schematic diagram of the external configuration of the image processing apparatus according to the present embodiment.

[0009] Figure 3 is a schematic diagram of the hardware configuration of the image processing apparatus according to this embodiment.

[0010] Figure 4 is a diagram for describing the direction in which the imaging unit moves relative to the scene in this embodiment.

[0011] Figure 5 It is a schematic diagram of the functional configuration of the processing unit in this embodiment.

[0012] Figure 6 It is a schematic diagram of the operation of the processing unit in this embodiment.

[0013] Figure 7 It is a schematic diagram of the operation of the processing unit in this embodiment.

[0014] Figure 8 It is a flowchart of the operation of the processing unit in this embodiment.

[0015] Figure 9 is a schematic diagram of the operation of the processing unit in the first modification example of the present embodiment.

[0016] Figure 10 is a schematic diagram of the operation of the processing unit in the first modification example of the present embodiment.

[0017] Figure 11 is a schematic diagram of the operation of the processing unit in the second modification example of the present embodiment.

[0018] Figure 12 is a schematic diagram of the operation of the processing unit in the third modification example of the present embodiment.

[0019] Figure 13 is a schematic diagram of the operation of the processing unit in the fourth modification example of the present embodiment.

[0020] Figure 14 is a schematic diagram of the operation of the processing unit in the fifth modification example of the present embodiment.

[0021] Figure 15 It is a flowchart illustrating the operation of the processing unit in the sixth modification example of this embodiment.

[0022] Figure 16 It is a flowchart illustrating the operation of the processing unit in the seventh modification example of this embodiment.

[0023] Figure 17 It is a schematic diagram of a use case of the learning library in the seventh modification example of this embodiment.

[0024] Figure 18 is a schematic diagram of another use case of the learning library in the seventh modification example of this embodiment.

[0025] Explanation of the reference numerals: 1-image processing device; 10, 20-imaging units; 30-processing unit; 30a-edge detection unit; 30b-determination unit; 30c-extraction unit; 30d-integration unit; 30e-display control unit; 30f-movement control unit; 40-display unit; 50-memory. DETAILED DESCRIPTION

[0026] Hereinafter, the image processing apparatus according to the present embodiment will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to this embodiment.

[0027] The image processing apparatus according to the present embodiment includes an event acquisition unit such as an event camera, and in the event that a scene change occurs, the event acquisition unit acquires the change of scene as an event, but is improved to appropriately acquire the event.

[0028] Figure 1 and Figure 2 2 is a schematic diagram of the external configuration of the image processing apparatus 1. The image processing apparatus 1 may be a portable electronic device, for example, a smartphone, a tablet computer, or the like.

[0029] When image processing device 1 is a portable electronic device, it is desirable for it to be thin and lightweight. Image processing device 1 may include a plate-shaped housing 2. Hereinafter, the short side of housing 2 is defined as the X direction, the long side of housing 2 is defined as the Y direction, and the direction perpendicular to the X and Y directions is defined as the Z direction.

[0030] The image processing apparatus 1 is configured to be able to image a subject and includes a plurality of imaging units 10 / 20 and a display unit 40 . Figure 1 shows the appearance viewed from the side opposite to the display unit 40 (ie, the +Z side), Figure 2 Then, the appearance viewed from the display unit 40 side (ie, the -Z side) is shown.

[0031] like Figure 1 As shown, the imaging unit 10 and the imaging unit 20 may be provided at positions adjacent to each other in the XY directions in the housing 2 .

[0032] The imaging unit 10 can acquire an image of a scene in frames. The imaging unit 10 is also referred to as a camera. The imaging unit 10 includes an optical system 11 and an imaging sensor 12. The optical system 11 is arranged so that the optical axis PA1 extends substantially along the Z direction and intersects with the imaging surface 12a of the corresponding imaging sensor 12. At least a portion of the optical system 11 is exposed from the housing 2. The optical system 11 forms an image of a subject on the imaging surface 12a of the imaging sensor 12. The imaging sensor 12 acquires an image of the scene in frames based on the object image formed on the imaging surface 12a. The image acquired in frames can be a color image. The image acquired in frames will be referred to as a frame image.

[0033] The imaging unit 20 can acquire events of the scene on the event pool. The imaging unit 20 is also called an event camera. The imaging unit 20 includes an optical system 21 and an imaging sensor 22. The optical system 21 is arranged so that the optical axis PA2 extends substantially along the Z direction and intersects with the imaging surface 22a of the corresponding imaging sensor 22. At least a portion of the optical system 21 is exposed from the housing 2. The optical system 21 forms an object image on the imaging surface 22a of the imaging sensor 22. The imaging sensor 22 acquires events of the scene on the event pool based on the object image formed on the imaging surface 22a. The event data obtained on the event pool can be constructed by image processing. The image constructed using the event pool will be referred to as an event image.

[0034] The optical axis PA1 of the optical system 11 and the optical axis PA2 of the optical system 21 are substantially parallel to each other. The viewing angle of the optical system 11 corresponds to the viewing angle of the optical system 21. Therefore, the movement of the imaging unit 10 relative to the scene corresponds to the movement of the imaging unit 20 relative to the scene. It should be noted that the imaging unit for acquiring an image of the scene and the imaging unit for acquiring events of the scene can be combined into a single imaging unit. For example, a single imaging unit can acquire an image of the scene in units of frames and acquire events of the scene on an event pool in parallel with the image acquisition, thereby outputting a plurality of pixel signals indicating the image and event data indicating the event, respectively.

[0035] like Figure 2 As shown, the display unit 40 is provided on the opposite side of the housing 2 from the imaging unit 10 and the imaging unit 20. The display unit 40 includes a display screen 41. For example, the display screen 41 extends in a substantially rectangular shape in the X and Y directions. In the image processing apparatus 1, images captured by the imaging unit 10 can be displayed on the display screen 41 of the display unit 40, and events captured by the imaging unit 20 can also be displayed on the display screen 41 of the display unit 40.

[0036] Figure 3 Schematic diagram of the hardware configuration of image processing device 1. Image processing device 1 also includes a processing unit 30, a display unit (i.e., display module) 40, a memory 50, a display input unit (i.e., input module) 60, a positioning unit (i.e., a global navigation satellite system (GNSS) module) 3, a communication unit (i.e., a wireless communication module) 4, a speaker 5, a microphone 6, a codec (CODEC) 7, an inertial measurement unit (IMU) 8, a bus 9, an antenna AT1, and an antenna AT2. Processing unit 30 includes an image signal processor (ISP) 31 and a processor 32. Image signal processor 31 performs processing primarily through hardware. Processor 32 performs processing primarily through software.

[0037] The image signal processor 31, display unit 40, memory 50, display input unit 60, positioning unit 3, communication unit 4, codec 7, inertial measurement unit 8, and processor 32 are connected via bus 9 so as to be able to communicate with each other. Antenna AT1 is connected to bus 9 via positioning unit 3. Antenna AT2 is connected to bus 9 via communication unit 4.

[0038] The imaging module 10 also includes a driver (i.e., an imaging sensor driver) 13, a sensor (i.e., a gyroscope (GYRO) sensor) 14, a driver (i.e., a focus and optical image stabilization (FOCUS & OIS) driver) 15, and an actuator (i.e., a FOCUS & OIS actuator) 16. The optical system 11 includes a lens 11a and a lens 11b. Lens 11b is movable in the direction of optical axis PA1. The optical axis PA1 is a direction along optical axis PA1.

[0039] The driver 15 performs focus adjustment control and / or optical image stabilizer (OIS) control under the control of the processing unit 30. In focus adjustment control, the driver 15 can move the lens 11b in the direction of the optical axis PA1 via the actuator 16 to focus the optical system 11. In addition, the sensor 14 can detect vibration applied to the optical system 11. In OIS control, the driver 15 can vibrate the lens 11b via the actuator 16 based on the detection result of the sensor 14 to eliminate the influence of the vibration applied to the optical system 11. The driver 15 is capable of vibrating the lens 11b in multiple directions (for example, 2 directions, 4 directions, 8 directions, 16 directions, etc.) intersecting the optical axis PA1 via the actuator 16.

[0040] The driver 13 drives the imaging sensor 12 under the control of the processing unit 30. The imaging sensor 12 includes a pixel array 12b. In the pixel array 12b, multiple pixels are arranged two-dimensionally. Each pixel includes a light receiving element. Each pixel performs photoelectric conversion in the light receiving element based on the received light, and generates and outputs a pixel signal. The imaging sensor 12 obtains multiple pixel signals indicating an image of a scene based on the object image formed in the pixel array 12b, and outputs these pixel signals to the processing unit 30.

[0041] The processing unit 30 receives the plurality of pixel signals from the imaging unit 10. The processing unit 30 processes the plurality of pixel signals to generate a frame image.

[0042] The imaging unit 20 also includes a driver (i.e., imaging sensor driver) 23, a sensor (i.e., GYRO sensor) 24, a driver (i.e., Focus & OIS driver) 25, an actuator (i.e., Focus & OIS actuator) 26, and an event detection circuit (i.e., Event detector). The optical system 21 includes a lens 21a and a lens 21b. The lens 21b can move in the direction of the optical axis PA2. The optical axis PA2 direction is the direction along the optical axis PA2.

[0043] The driver 25 performs focus adjustment control and / or OIS control under the control of the processing unit 30. In focus adjustment control, the driver 25 can move the lens 21b in the direction of the optical axis PA2 via the actuator 26 to focus the optical system 21. In addition, the sensor 24 can detect vibration applied to the optical system 21. In OIS control, the driver 25 can vibrate the lens 21b via the actuator 26 based on the detection result of the sensor 24 to eliminate the influence of the vibration applied to the optical system 21. The driver 25 can vibrate the lens 21b in multiple directions (for example, 2 directions, 4 directions, 8 directions, 16 directions, etc.) intersecting the optical axis PA2 via the actuator 26.

[0044] The driver 23 drives the imaging sensor 22 under the control of the processing unit 30. The imaging sensor 22 has a pixel array 22b. In the pixel array 22b, a plurality of pixels are arranged in a two-dimensional manner. Each pixel includes a light receiving unit and an event detection unit. Each pixel performs photoelectric conversion according to the received light by the light receiving unit, and the event detection unit detects whether a brightness change exceeding a threshold value occurs based on the result of the photoelectric conversion. When a brightness change exceeding the threshold value occurs, each pixel generates and outputs event data through the event detection unit. The imaging sensor 22 obtains a plurality of event data indicating events of the scene based on the object image formed in the pixel array 22b, and outputs the event data to the processing unit 30.

[0045] The processing unit 30 receives the above-mentioned multiple event data from the imaging unit 20. The processing unit 30 performs processing such as framing on the above-mentioned multiple event data to generate an event image. Event data is data generated in pixels where a brightness change exceeds a threshold value. This is mainly caused by the relative movement of the imaging unit 10 / 20 with respect to the object OB. This relative movement is caused by at least one of the movement of the object OB and the movement of the imaging unit 10 / 20 itself. The processing unit 30 can perform predetermined processing using the event image.

[0046] For example, Figure 4 As shown in (a), it is assumed that the imaging unit 10 images a scene including an object OB1 extending in a rod shape in the Y direction and obtains a frame image IM0. Figure 4is a diagram for describing the direction in which the imaging unit 20 moves relative to the scene. The processing unit 30 may display the following on the display screen of the display unit 40: Figure 4 (a) shows the frame image IM0.

[0047] In reality, if the scene is dark, the contrast difference between the background and the object OB1 in the frame image IM0 is relatively small. Therefore, it is difficult for the processing unit 30 to accurately perform object recognition, such as identifying the outline of the object OB1, using the frame image IM0. It is also difficult for the processing unit 30 to accurately recognize the motion of the object OB1 using the frame image IM0 and accurately perform deblurring to correct the image blur caused by the motion.

[0048] Assume that the imaging unit 10 / 20 is positioned relative to the object OB1 extending in a rod-like shape in the Y direction. Figure 4 In response to this, the imaging unit 20 acquires a plurality of pieces of event data EV1_1 to EV1_n corresponding to the outline of the object OB1 extending in the Y direction, as shown in FIG. Figure 4 (b) is shown. n is an integer of 2 or greater. The processing unit 30 can perform processing such as framing on the plurality of event data EV1_1 to EV1_n to generate Figure 4 (b) shows the event image EVIM1.

[0049] The plurality of pieces of event data EV1_1 to EV1_n in event image EVIM1 cover most of the outline of object OB1. Therefore, processing unit 30 can easily use event image EVIM1 to perform object recognition for identifying the outline of object OB1, etc. Processing unit 30 can easily use event image EVIM1 to identify the motion of object OB1 and perform deblurring to correct image blur caused by the motion.

[0050] At the same time, the imaging unit 10 obtains Figure 4 The frame image IM0 in (a) is similar to the frame image IM0, but it is assumed that the imaging unit 10 / 20 is positioned relative to the object OB1 extending in the shape of a rod in the Y direction. Figure 4 In response to this, the imaging unit 20 acquires a plurality of pieces of event data EV1_1 to EV1_k corresponding to the outline of the object OB1 extending in the X direction, as shown in FIG. Figure 4 (d) is shown. k is an integer of 2 or greater. The processing unit 30 can perform processing such as framing on the plurality of event data EV1_1 to EV1_k to generate Figure 4 (d) shows the event image EVIM2.

[0051] The pieces of event data EV2_1 to EV2_k in event image EVIM2 cover a small portion of the outline of object OB1. Therefore, it is difficult for processing unit 30 to use event image EVIM2 to perform object recognition, such as identifying the outline of object OB1. It is also difficult for processing unit 30 to use event image EVIM2 to identify the motion of object OB1 and perform deblurring to correct image blur caused by the motion.

[0052] like Figure 4 As shown, in order for the imaging unit 20 to properly capture events, it is assumed that there is an appropriate direction in which the imaging unit 20 moves relative to the scene. If the imaging unit 20 does not properly capture events and the amount of captured events is small, the accuracy of the predetermined processing using the events in the processing unit 30 may be reduced.

[0053] Therefore, the processing unit 30 detects edges in multiple directions from at least a portion of the frame image obtained by the imaging unit 10, determines the direction in which the imaging unit 20 should move relative to the scene based on the detection results, and prompts the user to move in that direction, thereby achieving high accuracy of predetermined processing using the events obtained by the imaging unit 20.

[0054] The processing unit 30 can be as follows Figure 5 Configuration shown. Figure 5 : is a schematic diagram of the functional configuration of the processing unit 30. The processing unit 30 includes an edge detection unit 30a, a determination unit 30b, and a display control unit 30e.

[0055] The edge detection unit 30a receives a frame image from the imaging unit 10. The edge detection unit 30a detects edges in multiple directions from at least a portion of the frame image. These multiple directions may be directions located in a plane perpendicular to the optical axis PA1 of the imaging unit 10 or directions located in a plane approximately perpendicular to the optical axis PA2 of the imaging unit 20. The edge detection unit 30a can detect edges in the entire frame image in multiple directions.

[0056] For example, after receiving the following information from the imaging unit 10: Figure 6 In the case of the frame image IM1 shown in (a), the edge detection unit 30a detects edges from the entire frame image IM1. Figure 6 3 is a schematic diagram of the operation of the processing unit 30. The edge detection unit 30a may detect edges from the entire frame image IM1 using a Sobel filter, a Canny filter, or the like, or may detect edges from the entire frame image IM1 using a method such as random sample consensus (RANSAC) or Hough transform.

[0057] The edge detection unit 30a generates the following according to the detected edge: Figure 6(b) shows edge information ED1. The edge information ED1 includes information about the pixel position of the edge in the frame image IM1. Figure 6 As shown in (b), the edge detection unit 30a can generate edge information ED1 as collected information of pixel positions of edges or as an edge image ED1. The edge image ED1 includes a line drawing pattern corresponding to the edge information ED1.

[0058] The edge detection unit 30a extracts edges in multiple directions from the edge information ED1. The number of directions in which edges are extracted is arbitrary, but can be as follows: Figure 7 (a) shows two directions DR1 / DR2. Direction DR1 is along the X direction, and direction DR2 is along the Y direction.

[0059] Or, as Figure 7 As shown in (b), the number of directions for extracting edges can be four directions DR1 to DR4. Direction DR3 can be a direction obtained by rotating the X direction counterclockwise by approximately 45 degrees. Direction DR4 can be a direction obtained by rotating the X direction counterclockwise by approximately 135 degrees.

[0060] Alternatively, the number of directions to extract edges can be as Figure 7 (c) shows eight directions DR1 to DR8. Direction DR5 may be a direction obtained by rotating the X direction counterclockwise by approximately 22.5°. Direction DR6 may be a direction obtained by rotating the X direction counterclockwise by approximately 67.5°. Direction DR7 may be a direction obtained by rotating the X direction counterclockwise by approximately 112.5°. Direction DR8 may be a direction obtained by rotating the X direction counterclockwise by approximately 158.5°.

[0061] Or, as Figure 7 As shown in (d), the number of directions for extracting edges may be 16 directions DR1 to DR16. Direction DR9 may be a direction obtained by rotating the X direction counterclockwise by approximately 11.25°. Direction DR10 may be a direction obtained by rotating the X direction counterclockwise by approximately 33.75°. Direction DR11 may be a direction obtained by rotating the X direction counterclockwise by approximately 56.25°. Direction DR12 may be a direction obtained by rotating the X direction counterclockwise by approximately 78.75°. Direction DR13 may be a direction obtained by rotating the X direction counterclockwise by approximately 101.25°. Direction DR14 may be a direction obtained by rotating the X direction counterclockwise by approximately 123.75°. Direction DR15 may be a direction obtained by rotating the X direction counterclockwise by approximately 146.25°. Direction DR16 may be a direction obtained by rotating the X direction counterclockwise by approximately 168.75°.

[0062] The edge detection unit 30a generates an edge detection result based on the edge extraction results in each of the multiple directions. The edge detection unit 30a can count the number of pixels included in the edge in each of the multiple directions to obtain the number of pixels in the edge and generate the edge detection result. The edge detection result includes information for identifying the direction and information about the number of pixels in the edge in the multiple directions, which is associated with the information for identifying the direction.

[0063] The determining unit 30b receives the edge detection result from the edge detecting unit 30a and determines the direction in which the imaging unit 20 moves relative to the scene based on the edge detection result.

[0064] Based on the edge detection result, the determination unit 30b determines the direction in which events of the scene acquired by the imaging unit 20 are appropriately acquired when the imaging unit 20 moves relative to the scene as the direction in which the imaging unit 20 moves. Based on the edge detection result, the determination unit 30b can determine the direction in which the number of events of the scene acquired by the imaging unit 20 is the largest when the imaging unit 20 moves relative to the scene as the direction in which the imaging unit 20 moves. The determination unit 30b can determine, from among a plurality of directions, a direction that intersects with the direction in which the number of pixels of the edge detected in the entire frame image is the largest as the direction in which the imaging unit 20 moves.

[0065] Based on the edge detection result, the determination unit 30b can compare the number of pixels of edges in multiple directions and specify the direction with the largest number of pixels in the multiple directions. The determination unit 30b can determine the specified direction as the direction in which the imaging unit 20 moves.

[0066] For example, the edge detection unit 30a generates Figure 6 In the case of edge information ED1 shown in (b), the number of pixels of the edge in direction DR2 in the edge detection result may be the largest. In response to this, the determination unit 30b can determine the direction DR1 intersecting (for example, orthogonal to) the direction DR2 as the direction in which the imaging unit 20 moves.

[0067] Note that in direction DR1 (refer to Figure 7 When the number of pixels of the edge on (a) is the largest, the determination unit 30 b may determine the direction DR2 intersecting (for example, orthogonal) with the direction DR1 as the direction in which the imaging unit 20 moves.

[0068] Alternatively, in direction DR3 (refer to Figure 7 In the case where the number of pixels of the edge on (b)) is the largest, the determination unit 30b may determine a direction DR4 intersecting (eg, orthogonal to) the direction DR3 as the direction in which the imaging unit 20 moves.

[0069] Alternatively, in direction DR5 (refer to Figure 7 In the case where the number of pixels of the edge on (c)) is the largest, the determination unit 30b may determine a direction DR7 intersecting (eg, orthogonal to) the direction DR5 as the direction in which the imaging unit 20 moves.

[0070] Alternatively, in direction DR16 (see Figure 7 When the number of pixels of the edge on (d)) is the largest, the determination unit 30b may determine the direction DR12 intersecting (eg, orthogonal to) the direction DR16 as the direction in which the imaging unit 20 moves.

[0071] The determination unit 30b supplies the determination result of the moving direction to the display control unit 30e.

[0072] The display control unit 30e receives the frame image IM1 from the imaging unit 10. The display control unit 30e receives the result of the determination of the movement direction from the determination unit 30b. The display control unit 30e can display the frame image IM1 on the display unit 40 in a display mode that can identify the direction in which the imaging unit 20 is moving, based on the result of the determination of the movement direction. The display control unit 30e can display a display object indicating the direction in which the imaging unit 20 is moving as a superposition on the frame image IM1.

[0073] For example, the edge detection unit 30a generates Figure 6 (b) shows the edge information ED1, and the determination unit 30b determines that the direction DR1 is the direction in which the imaging unit 20 moves, the display control unit 30e may display the edge information ED1 as shown in FIG. Figure 6 The display object DO1 shown in (c) is superimposed on the frame image IM1 and displayed on the display screen 41 of the display unit 40. The display object DO1 may have an arrow shape indicating the moving direction DR1.

[0074] Note that the direction DR2 is determined by the determination unit 30b (see Figure 7 When (a) is the direction in which the imaging unit 20 moves, the display control unit 30e may superimpose a display object DO2 on the frame image IM1 and display it on the display screen 41 of the display unit 40. The display object DO2 may have an arrow shape indicating the moving direction DR2.

[0075] Alternatively, the direction DR4 is determined in the determination unit 30b (see Figure 7 When (b)) is the direction in which the imaging unit 20 moves, the display control unit 30e may superimpose a display object DO4 on the frame image IM1 and display it on the display screen 41 of the display unit 40. The display object DO4 may have an arrow shape indicating the moving direction DR4.

[0076] Alternatively, the direction DR7 is determined in the determination unit 30b (see Figure 7 (c)) is the direction in which the imaging unit 20 moves, the display control unit 30e may superimpose a display object DO7 on the frame image IM1 and display it on the display screen 41 of the display unit 40. The display object DO7 may have an arrow shape indicating the moving direction DR7.

[0077] Alternatively, the direction DR12 is determined in the determination unit 30b (see Figure 7 When (d)) is the direction in which the imaging unit 20 moves, the display control unit 30e may superimpose a display object DO12 on the frame image IM1 and display it on the display screen 41 of the display unit 40. The display object DO12 may have an arrow shape indicating the moving direction DR12.

[0078] Therefore, a user who is holding the housing 2 of the image processing apparatus 1 and viewing the display screen 41 of the display unit 40 can be prompted to move the housing 2 in the direction DR indicated by the display object DO. In response, when the user moves the housing 2 in the direction DR, the imaging unit 20 moves in the direction DR relative to the scene, thereby effectively increasing the amount of events captured by the imaging unit 20.

[0079] Next, we will refer to Figure 8 The operation of the image processing apparatus 1 is described. Figure 8 It is a flowchart of the operation of the image processing apparatus 1 .

[0080] In the image processing apparatus 1, when acquiring a frame image (i.e., step S1), the imaging unit 10 supplies the frame image to the edge detection unit 30a. The edge detection unit 30a detects edges from the entire region of the frame image (i.e., step S2) and generates edge information ED. The edge information ED includes information about the pixel positions of edges in the frame image IM1.

[0081] The edge detection unit 30a extracts information about edges in each direction (i.e., step S3). The edge detection unit 30a selects a direction DR from the plurality of directions DR in which edges are to be extracted, and extracts edge information in the selected direction DR from the edge information ED. The edge detection unit 30a extracts the edge in the selected direction DR from the edge information ED and counts the number of pixels included in the extracted edge, thereby obtaining the number of pixels of the edge in the selected direction DR. The edge detection unit 30a provides the number of pixels of the edge as an edge detection result associated with the selected direction DR to the determination unit 30b.

[0082] After obtaining the number of pixels of the edge in the selected direction DR, the edge detection unit 30a determines whether there is an unselected direction DR among the multiple directions to be extracted edges, and repeats the above step S3 until all directions in the multiple directions DR are processed (i.e., "No" in step S4).

[0083] When the edge detection unit 30a processes all of the multiple directions (ie, "yes" in step S4), the determination unit 30b compares the information of the edge detection results in all directions (ie, step S5). The determination unit 30b compares the number of pixels of the edge in the multiple directions DR.

[0084] Based on the comparison result of step S5, the determination unit 30b determines the direction DR in which the imaging unit 20 moves relative to the scene (i.e., step S6). The determination unit 30b specifies the direction DR in which the number of pixels of the edge detected in the entire frame image is the largest among the multiple directions DR, and determines the direction intersecting (e.g., orthogonal to) the specified direction DR as the direction DR in which the imaging unit 20 moves relative to the scene.

[0085] The display control unit 30e displays the frame image IM1 on the display unit 40 in a display format that allows identification of the direction DR in which the imaging unit 20 is moving (i.e., step S7). The display control unit 30e receives the frame image IM1 from the imaging unit 10. The display control unit 30e receives the result of determining the direction of movement DR from the determination unit 30b. Based on the result of determining the direction DR in which the imaging unit 20 is moving, the display control unit 30e generates a display object DO indicating the direction DR in which the imaging unit 20 is moving. The display control unit 30e superimposes the display object DO indicating the direction DR in which the imaging unit 20 is moving on the frame image IM1 displayed on the display unit 40.

[0086] As described above, in this embodiment, in the processing unit 30 of the image processing device 1, the edge detection unit 30a detects edges in multiple directions DR from the entire frame image acquired by the imaging unit 10, the determination unit 30b determines the direction in which the imaging unit 20 should move based on the detection results, and the display control unit 30e displays a display object DO on the display unit 40 that prompts movement in the determined direction DR. This encourages a user holding the housing 2 of the image processing device 1 and viewing the display unit 40 to move the housing 2 in the direction DR. In response, when the user moves the housing 2 in the direction DR, the imaging unit 20 moves in the direction DR relative to the scene, increasing the number of events acquired by the imaging unit 20. Consequently, in the image processing device 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0087] Note that, as a first modification of the present embodiment, the image processing apparatus 1 may control the imaging unit 10 / 20 to move in the determined direction DR, rather than performing movement of the display cue in the determined direction DR.

[0088] In this case, the processing unit 30 includes a movement control unit 30f (see Figure 5 ) instead of the display control unit 30e. The movement control unit 30f receives the determination result of the movement direction DR from the determination unit 30b. The movement control unit 30f can control the actuator 16 / 26 to move the lens 11b / 21b of the optical system 11 / 21 in the movement direction DR based on the determination result of the movement direction DR. The movement control unit 30f can vibrate the lens 11b / 21b of the optical system 11 / 21 once in the movement direction DR, or can vibrate the lens 11b / 21b of the optical system 11 / 21 multiple times in the movement direction DR.

[0089] For example, assuming that the edge detection unit 30a is based on Figure 9 The frame image IM1 shown in (a) is generated Figure 9 In the edge information ED1 shown in (b) (e.g., in the form of edge image EIM1), the determination unit 30b specifies that the number of pixels of the edge in direction DR2 is the largest. The determination unit 30b determines the direction DR1 that intersects (e.g., is orthogonal to) direction DR2 as the direction in which the imaging unit 20 moves. Figure 9 FIG. 3 is a diagram illustrating the operation of the processing unit 30 in the first modification of the present embodiment. Based on the result of the determination of the direction DR1, the movement control unit 30f can control the actuator 16 / 26 to move in the following manner: Figure 9 The lens 11b / 21b of the optical system 11 / 21 is moved in the direction DR1 indicated by the dashed arrow in (c).

[0090] Alternatively, assume that the determining unit 30b determines that the number of pixels of the edge is Figure 10 The maximum value is in the direction DR1 shown in FIG. 3( a). The determination unit 30 b determines the direction DR2 intersecting (e.g., orthogonal to) the direction DR1 as the direction in which the imaging unit 20 moves. The movement control unit 30 f can control the actuator 16 / 26 to move the lens 11b / 21b of the optical system 11 / 21 in the direction DR2 based on the determination result of the direction DR2.

[0091] Alternatively, assume that the determining unit 30b determines that the number of pixels of the edge is Figure 10(b) is the largest in direction DR3. The determination unit 30b determines a direction DR4 that intersects (e.g., is orthogonal to) the direction DR3 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move the lens 11b / 21b of the optical system 11 / 21 in the direction DR4 based on the determination result of the direction DR4.

[0092] Alternatively, assume that the determining unit 30b determines that the number of pixels of the edge is Figure 10 (c) is the maximum in direction DR5. The determination unit 30b determines a direction DR7 that intersects (e.g., is orthogonal to) the direction DR5 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move the lens 11b / 21b of the optical system 11 / 21 in the direction DR7 based on the determination result of the direction DR7.

[0093] Alternatively, assume that the determining unit 30b determines that the number of pixels of the edge is Figure 10 (d) is the maximum in direction DR16 shown in FIG. The determination unit 30b determines the direction DR12 intersecting (e.g., orthogonal to) the direction DR16 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move the lens 11b / 21b of the optical system 11 / 21 in the direction DR12 based on the determination result of the direction DR12.

[0094] Therefore, since the lens 21b of the optical system 21 moves in a certain direction DR relative to the scene, the number of events acquired by the imaging unit 20 can be increased. Therefore, in the image processing device 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0095] Alternatively, the movement control unit 30 f may control the actuator 16 / 26 to move the imaging sensor 12 / 22 in the movement direction DR according to the result of determining the movement direction DR. The movement control unit 30 f may vibrate the imaging sensor 12 / 22 once in the movement direction DR, or may vibrate the imaging sensor 12 / 22 a plurality of times in the movement direction DR.

[0096] For example, assuming that the image receiving unit 10 receives the following Figure 9 The edge detection unit 30a generates the frame image IM1 shown in (a) as follows: Figure 9 (b) shows the edge information ED1, and the determination unit 30a determines the direction DR1 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move in the direction as shown in FIG. Figure 9 The imaging sensor 12 / 22 is moved in the direction DR1 indicated by the dot-dash arrow in (c).

[0097] Alternatively, assume that the determining unit 30b determines that the number of pixels of the edge is Figure 10 (e) is the largest in direction DR1. The determination unit 30b determines a direction DR2 that intersects (e.g., is orthogonal to) the direction DR1 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move the imaging sensor 12 / 22 in the direction DR2 based on the determination result of the direction DR2.

[0098] Alternatively, assume that the determining unit 30b determines that the number of pixels of the edge is Figure 10 The maximum value is in the direction DR3 shown in FIG. The determination unit 30 b determines the direction DR4 that intersects (e.g., is orthogonal to) the direction DR3 as the direction in which the imaging unit 20 moves. The movement control unit 30 f can control the actuator 16 / 26 to move the imaging sensor 12 / 22 in the direction DR4 based on the determination result of the direction DR4.

[0099] Alternatively, assume that the determining unit 30b determines that the number of pixels of the edge is Figure 10 (g) is the largest in the direction DR5 shown. The determination unit 30b determines the direction DR7 that intersects (e.g., is orthogonal to) the direction DR5 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move the imaging sensor 12 / 22 in the direction DR7 based on the determination result of the direction DR7.

[0100] Alternatively, assume that the determining unit 30b determines that the number of pixels of the edge is Figure 10 (h) is the largest in direction DR16. The determination unit 30b determines the direction DR12 that intersects (e.g., is orthogonal to) the direction DR16 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move the imaging sensor 12 / 22 in the direction DR12 based on the determination result of the direction DR12.

[0101] Thus, since the imaging sensor 22 moves in a certain direction DR relative to the scene, the number of events acquired by the imaging unit 20 can be increased. Therefore, in the image processing apparatus 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0102] Furthermore, as a second modification of the present embodiment, the image processing apparatus 1 may determine the direction intersecting the direction in which the sum of the lengths of the edges is the longest as the direction of movement, rather than determining the direction intersecting the direction in which the number of pixels of the edge is the largest, as the direction of movement, and may perform display prompt movement in the determined direction. For example, in the case where there is a direction in which the length of each edge element (e.g., a line segment) is long, even when the number of pixels of the edge is small, it is expected that the amount of events that occur in the imaging unit 20 when movement is performed in the direction intersecting the direction in which the sum of the lengths of the edges is long will be greater than when movement is performed in the direction intersecting the direction in which the number of pixels of the edge is large.

[0103] In this case, the determination unit 30b in the processing unit 30 can determine the direction that intersects (for example, is orthogonal) with the direction in which the sum of the lengths of the edges detected in the entire frame image is the longest among the multiple directions in which the edge detection unit 30a detects the edge as the direction in which the imaging unit 20 moves.

[0104] The edge detection unit 30a extracts edges in each direction from the edge information ED, obtains the length of each of the multiple edge elements (e.g., multiple line segments) included in the extracted edge through a Hough transform, etc., and adds the lengths of the multiple edge elements to obtain the sum of the edge lengths. The determination unit 30b can compare the sum of the edge lengths in multiple directions and specify the direction in which the sum of the edge lengths in the multiple directions is the longest. The determination unit 30b can determine the specified direction as the direction in which the imaging unit 20 is moving.

[0105] For example, the edge detection unit 30a is based on the following Figure 11 The frame image IM2 shown in (a) is generated as follows Figure 11 In the case of edge information ED2 (eg, in the form of edge image EIM2) shown in (b), the sum of the lengths of the edges in the direction DR2 in the edge detection result may be the longest. Figure 11 is a schematic diagram of the operation of the processing unit 30 in the second modification of the present embodiment. The determination unit 30b determines the direction DR1 intersecting (for example, orthogonal to) the direction DR2 as the direction in which the imaging unit 20 moves. The display control unit 30e can Figure 11 The display object DO1a shown in (c) is superimposed on the frame image IM2 and displayed on the display screen 41 of the display unit 40 according to the determination result of the moving direction DR1. The display object DO1a may have an arrow shape indicating the moving direction DR1.

[0106] Therefore, a user holding the housing 2 of the image processing device 1 and viewing the display screen 41 of the display unit 40 can be encouraged to move the housing 2 in the direction DR indicated by the display object DO1a. In response, when the user moves the housing 2 in the direction DR, the imaging unit 20 moves in the direction DR relative to the scene, effectively increasing the number of events captured by the imaging unit 20. Therefore, in the image processing device 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0107] In addition, as a third modification of this embodiment, the image processing device 1 can determine the direction that intersects with the direction in which the sum of the lengths of the edges is the longest as the moving direction, instead of determining the direction that intersects with the direction in which the number of pixels of the edges is the largest as the moving direction, and control the imaging unit 10 / 20 to move in the determined direction.

[0108] In this case, the determination unit 30b in the processing unit 30 can determine the direction that intersects (for example, is orthogonal) with the direction in which the sum of the lengths of the edges detected in the entire frame image is the longest among the multiple directions in which the edge detection unit 30a detects the edge as the direction in which the imaging unit 20 moves.

[0109] The processing unit 30 includes a movement control unit 30f (see Figure 5 ) instead of the display control unit 30e. The movement control unit 30f receives the determination result of the movement direction DR from the determination unit 30b. The movement control unit 30f can control the actuator 16 / 26 to move the lens 11b / 21b of the optical system 11 / 21 in the movement direction DR based on the determination result of the movement direction DR. The movement control unit 30f can vibrate the lens 11b / 21b of the optical system 11 / 21 once in the movement direction DR, or can vibrate the lens 11b / 21b of the optical system 11 / 21 multiple times in the movement direction DR.

[0110] For example, it is assumed that the edge detection unit 30a is based on the following Figure 12 The frame image IM2 shown in (a) is generated as follows Figure 12 The edge information ED2 shown in (b) (e.g., in the form of an edge image EIM2) is obtained, and the determination unit 30b determines that the sum of the lengths of the edges in the direction DR2 is the largest. The determination unit 30b determines the direction DR1 that intersects (e.g., is orthogonal to) the direction DR2 as the direction in which the imaging unit 20 is moving. Figure 12 FIG. 3 is a schematic diagram of the operation of the processing unit 30 in the third modified example of the present embodiment. Based on the result of the determination of the direction DR1, the movement control unit 30f can control the actuator 16 / 26 to move in the following manner: Figure 12 The lens 11b / 21b of the optical system 11 / 21 is moved in the direction DR1 indicated by the dashed arrow in (c).

[0111] In this way, since the lens 21b of the optical system 21 moves in a certain direction DR relative to the scene, the number of events captured by the imaging unit 20 can be effectively increased. Therefore, in the image processing device 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0112] Alternatively, the movement control unit 30f may control the actuator 16 / 26 to move the imaging sensor 12 / 22 in the movement direction DR based on the result of determining the movement direction DR. The movement control unit 30f may vibrate the imaging sensor 12 / 22 in the movement direction DR once or multiple times.

[0113] For example, it is assumed that the edge detection unit 30a is based on the following Figure 12 The frame image IM2 shown in (a) is generated as follows Figure 12 (b) shows edge information ED2 (e.g., in the form of edge image EIM2), and the determination unit 30b determines that the sum of the lengths of the edges in the direction DR2 is the largest. The determination unit 30b determines the direction DR1 intersecting (e.g., orthogonal to) the direction DR2 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move in the direction DR1 according to the determination result of the direction DR1. Figure 12 The imaging sensor 12 / 22 is moved in the direction DR1 indicated by the dot-dash arrow in (c).

[0114] Thus, since the imaging sensor 22 moves in a certain direction DR relative to the scene, the amount of events acquired by the imaging unit 20 can be effectively increased. Therefore, in the image processing device 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0115] Furthermore, as a fourth modification of this embodiment, the image processing device 1 may detect edges from a partial area of ​​the frame image, rather than detecting edges from the entire frame image, and may determine and display the edges based on the detection results. For example, when uniform edges exist in the background portion depending on the scene, and the edges are detected from the entire image, if many edges are detected in a direction different from the main edge of the main subject, there is a possibility that the event of the main subject may not have occurred sufficiently even when the imaging unit 20 moves in a direction intersecting with that direction.

[0116] In this case, the processing unit 30 further includes an extraction unit 30c (see Figure 5 ). The extraction unit 30c extracts the main object area from the frame image. The main object area is an area including the main object. Any method can be used as a method for extracting the main object area.

[0117] For example, the extraction unit 30c extracts multiple feature points from the frame image and specifies a region including one or more feature points with significant features in the frame image based on the color correlation of the extracted feature points. Thus, the extraction unit 30c can extract the main object region in the frame image.

[0118] Alternatively, the extraction unit 30c can perform segmentation using a learning library of machine learning. The learning library has a neural network that includes one or more intermediate layers between an input layer and an output layer. The learning library is pre-learned using an image including the main object region, and the weights in each node of the input layer, one or more intermediate layers, and the output layer are adjusted to appropriately output the main object region. The learning library with the adjusted weights can be stored in the memory 50 (see Figure 3 ). The extraction unit 30c reads the learning library from the memory 50, inputs the frame image into the input layer of the learning library, and extracts the recognition result of the main object area from the output layer of the learning library. Therefore, the extraction unit 30c can extract the main object area in the frame image.

[0119] Alternatively, the extraction unit 30c may divide the frame image into a plurality of regions, estimate the distance of each region using distance determination or the like for each of the plurality of regions, and designate a region from the plurality of regions where the estimated value falls within a predetermined distance range as the main subject region. Thus, the extraction unit 30c can extract the main subject region from the frame image.

[0120] The extraction unit 30c provides the extraction result of the main subject area to the edge detection unit 30a. Based on the extraction result of the main subject area, the edge detection unit 30a extracts the edge of the main subject area from the edge information ED for each of the multiple directions DR. The edge detection unit 30a counts the number of pixels included in the extracted edge for each of the multiple directions DR to obtain the number of pixels in the edge and generates an edge detection result. The edge detection unit 30a provides the edge detection result to the determination unit 30b. The determination unit 30b may determine the direction of movement of the imaging unit 20 as the direction intersecting the direction with the largest number of pixels of the edge detected in the main subject area among the multiple directions DR.

[0121] For example, in the extraction unit 30c, Figure 13 The main object area MO3 is extracted from the frame image IM3 shown in (a), and the edge detection unit 30a generates Figure 13 In the case of edge information ED3 (for example, in the form of edge image EIM3) on the main object area MO3 shown in (b), the number of pixels of the edge in the direction DR2 in the edge detection result can be the largest. Figure 13FIG. 3 is a diagram illustrating the operation of the processing unit 30 in the fourth modification of the present embodiment. The determination unit 30b determines the direction DR1 intersecting (eg, orthogonal to) the direction DR2 as the direction in which the imaging unit 20 moves. The display control unit 30e may display the direction DR1 as shown in FIG. Figure 13 The display object DO1b shown in (c) is superimposed on the frame image IM3 and displayed on the display screen 41 of the display unit 40 according to the determination result of the movement direction DR1 of the frame image IM2. The display object DO1b may have an arrow shape indicating the movement direction DR1.

[0122] Therefore, the user who is holding the housing 2 of the image processing device 1 and viewing the display screen 41 of the display unit 40 can be prompted to move the housing 2 in the direction DR indicated by the display object DO1b. In response, when the user moves the housing 2 in the direction DR, the imaging unit 20 moves in the direction DR relative to the scene, thereby effectively increasing the amount of events of the main object captured by the imaging unit 20.

[0123] Alternatively, the determination unit 30 b may determine, as the direction in which the imaging unit 20 moves, a direction intersecting with a direction in which the sum of the lengths of the edges detected in the main object area is longest among the plurality of directions DR.

[0124] For example, in the extraction unit 30c, Figure 13 The main object area MO3 is extracted from the frame image IM3 shown in (a), and the edge detection unit 30a generates Figure 13 In the case of edge information ED3 (for example, in the form of edge image EIM3) about the main object area MO3 shown in (b), the sum of the lengths of the edges in the direction DR2 in the edge detection result may be the longest. Figure 13 FIG. 3 is a diagram illustrating the operation of the processing unit 30 in the fourth modification of the present embodiment. The determination unit 30b determines the direction DR1 intersecting (eg, orthogonal to) the direction DR2 as the direction in which the imaging unit 20 moves. The display control unit 30e may display the direction DR1 as shown in FIG. Figure 13 The display object DO1b shown in (c) is superimposed on the frame image IM3 and displayed on the display screen 41 of the display unit 40 according to the determination result of the movement direction DR1. The display object DO1b may have an arrow shape indicating the movement direction DR1.

[0125] Therefore, the user who is holding the housing 2 of the image processing device 1 and viewing the display screen 41 of the display unit 40 can be prompted to move the housing 2 in the direction DR indicated by the display object DO1b. In response, when the user moves the housing 2 in the direction DR, the imaging unit 20 moves in the direction DR relative to the scene, thereby effectively increasing the amount of events of the main object captured by the imaging unit 20.

[0126] Furthermore, as a fifth modification of the present embodiment, the image processing apparatus 1 may detect edges from a partial region of a frame image, instead of detecting edges from the entire frame image, and perform determination and control based on the detection result.

[0127] In this case, the processing unit 30 includes a movement control unit 30f (see Figure 5 ) instead of the display control unit 30e, and further includes an extraction unit 30c (see Figure 5 ). The extraction unit 30c extracts the main object area from the frame image. The main object area is an area including the main object. The extraction unit 30c provides the extraction result of the main object area to the edge detection unit 30a. The edge detection unit 30a extracts the edge of the main object area from the edge information ED for each of the multiple directions DR based on the extraction result of the main object area, obtains the number of pixels of the extracted edge, and generates an edge detection result. The edge detection unit 30a provides the edge detection result to the determination unit 30b. The determination unit 30b can determine the direction that intersects with the direction with the largest number of pixels of the edge detected in the main object area among the multiple directions DR as the direction in which the imaging unit 20 moves. The movement control unit 30f receives the determination result of the movement direction DR from the determination unit 30b. The movement control unit 30f can control the actuator 16 / 26 to move the lens 11b / 21b of the optical system 11 / 21 in the movement direction DR based on the determination result of the movement direction DR.

[0128] For example, in the extraction unit 30c, Figure 14 The main object area MO3 is extracted from the frame image IM3 shown in (a), and the edge detection unit 30a generates Figure 14 In the case of edge information ED3 (eg, in the form of edge image EIM3) on the main object area MO3 shown in (b), the number of pixels of the edge in the direction DR2 in the edge detection result may be the largest. Figure 14 FIG. 5 is a diagram illustrating the operation of the processing unit 30 in the fifth modification of the present embodiment. The determination unit 30b determines the direction DR1 intersecting (e.g., orthogonal to) the direction DR2 as the direction in which the imaging unit 20 moves. Based on the determination result of the direction DR1, the movement control unit 30f can control the actuator 16 / 26 to move in the following manner: Figure 14 The lens 11b / 21b of the optical system 11 / 21 is moved in the direction DR1 indicated by the dashed arrow in (c).

[0129] In this manner, since the lens 21 b of the optical system 21 moves in a certain direction DR relative to the scene, the amount of events of the main subject acquired by the imaging unit 20 can be effectively increased.

[0130] Alternatively, the determination unit 30 b may determine, as the direction in which the imaging unit 20 moves, a direction intersecting with a direction in which the sum of the lengths of the edges detected in the main object area is longest among the plurality of directions DR.

[0131] For example, in the extraction unit 30c, Figure 14 The main object area MO3 is extracted from the frame image IM3 shown in (a), and the edge detection unit 30a generates Figure 14 In the case of edge information ED3 (e.g., in the form of edge image EIM3) about the main object area MO3 shown in (b), the sum of the lengths of the edges in the direction DR2 in the edge detection result can be the longest. The determination unit 30b determines the direction DR1 intersecting (e.g., orthogonal to) the direction DR2 as the direction in which the imaging unit 20 moves. The movement control unit 30f can control the actuator 16 / 26 to move in the direction DR1 according to the determination result of the direction DR1. Figure 14 The imaging sensor 12 / 22 is moved in the direction DR1 indicated by the dot-dash arrow in (c).

[0132] In this manner, since the imaging sensor 22 moves in a determined direction DR relative to the scene, the amount of events of the main subject acquired by the imaging unit 20 can be effectively increased.

[0133] Furthermore, as a sixth modification of this embodiment, the image processing device 1 can utilize multiple determination methods illustrated in this embodiment and its first through fifth modifications in conjunction with the edge detection results. These multiple determination methods can be performed independently of one another, and the image processing device 1 can utilize a single method from the multiple determination methods illustrated in this embodiment and its first through fifth modifications, or can utilize a combination of two or more of these methods. When utilizing a combination of two or more methods, the image processing device 1 combines the results of the individual methods based on a predetermined ratio to obtain a final result. This ratio can be determined in any manner, for example, manually or by calculation using machine learning or the like.

[0134] In this case, the processing unit 30 further includes an integration unit 30d (see Figure 5 When multiple determination methods are used in combination, the determination unit 30b determines the direction of movement of the imaging unit 20 based on multiple criteria and generates a determination result. The determination results based on the multiple criteria include one or more directions as the direction of movement of the imaging unit 20, and may include multiple different directions. The determination unit 30b provides the determination results based on the multiple criteria to the integration unit 30d. The integration unit 30d integrates the one or more directions determined by the determination unit 30b based on the multiple criteria to obtain the direction of movement of the imaging unit 20.

[0135] In this case, the integration unit 30d may output all of the one or more directions determined by the determination unit 30b as the direction of movement of the imaging unit 20. Alternatively, the integration unit 30d may prioritize the one or more directions determined by the determination unit 30b according to a predetermined rule and output the direction with the highest priority as the direction of movement of the imaging unit 20. Alternatively, the integration unit 30d may evaluate the one or more directions determined by the determination unit 30b while weighting multiple criteria and output the direction with the highest evaluation value as the direction of movement of the imaging unit 20. Alternatively, the integration unit 30d may average the one or more directions determined by the determination unit 30b and output the average direction as the direction of movement of the imaging unit 20.

[0136] For example, the image processing apparatus 1 may be as follows Figure 15 Proceed as shown. Figure 15 1 is a flowchart illustrating the operation of the image processing apparatus 1 according to the sixth modification example of the present embodiment.

[0137] In the image processing device 1, after executing step S1 and step S2, the edge detection unit 30a selects one direction DR from multiple directions DR to extract edges, and performs the processing of step S13, the processing of S14, the processing of steps S15 and S16, and the processing of steps S17 and S18 on the selected direction DR in parallel.

[0138] In step S13, information regarding the number of pixels included in the edge in the selected direction DR is extracted for the entire frame image. Edge detection unit 30a extracts the edge in the selected direction DR from edge information ED and counts the number of pixels included in the extracted edge, thereby obtaining the number of pixels in the edge in the selected direction DR. Edge detection unit 30a provides the number of pixels in the edge in the entire frame image as an edge detection result associated with the direction DR to determination unit 30b.

[0139] In step S14, information regarding the length of edges in the selected direction DR is extracted for the entire frame image. The edge detection unit 30a extracts the edges in the selected direction DR from the edge information ED, obtains the length of each of the multiple edge elements (e.g., multiple line segments) included in the extracted edges through a Hough transform or the like, and obtains the sum of the edge lengths by adding the lengths of the multiple edge elements. The edge detection unit 30a provides the sum of the edge lengths in the entire frame image as the edge detection result associated with the direction DR to the determination unit 30b.

[0140] In steps S15 and S16, information regarding the number of pixels included in the edge in the selected direction DR is extracted for the main object area. In step S15, the extraction unit 30c extracts the main object area from the frame image and provides the extraction result of the main object area to the edge detection unit 30a. Based on the extraction result of the main object area, the edge detection unit 30a extracts the edge of the main object area from the edge information ED in the selected direction DR. In step S16, the edge detection unit 30a counts the number of pixels included in the extracted edge to obtain the number of pixels of the edge in the selected direction DR. The edge detection unit 30a provides the number of pixels of the edge in the main object area as the edge detection result associated with the direction DR to the determination unit 30b.

[0141] In steps S17 and S18, information regarding the length of the edge in the selected direction DR is extracted for the main object area. In step S17, the extraction unit 30c extracts the main object area from the frame image and provides the extraction result of the main object area to the edge detection unit 30a. Based on the extraction result of the main object area, the edge detection unit 30a extracts the edge of the main object area from the edge information ED in the selected direction DR. In step S18, the edge detection unit 30a obtains the length of each of the multiple edge elements (e.g., multiple line segments) included in the extracted edge through Hough transform or the like, and obtains the sum of the edge lengths by, for example, adding the lengths of the multiple edge elements. The edge detection unit 30a provides the sum of the edge lengths in the main object area as the edge detection result associated with the direction DR to the determination unit 30b.

[0142] In response to the completion of all of the processing in step S13, the processing in step S14, the processing in steps S15 and S16, and the processing in steps S17 and S18, the edge detection unit 30a determines whether there is an unselected direction DR among the multiple directions in which the edge is to be extracted. The edge detection unit 30a repeats the parallel processing of the processing in step S13, the processing in step S14, the processing in steps S15 and S16, and the processing in steps S17 and S18 until all of the multiple directions DR have been processed (i.e., "No" in step S19). The edge detection unit 30a provides the number of pixels of the edge in association with the direction DR to the determination unit 30b.

[0143] When the edge detection unit 30a processes all of the multiple directions (ie, "Yes" in step S19), the determination unit 30b compares information of edge detection results in all directions (ie, step S20).

[0144] The determination unit 30b compares the number of pixels of the edges of the entire frame image in a plurality of directions DR.

[0145] The determination unit 30b compares the sum of the lengths of the edges of the entire frame image in a plurality of directions DR.

[0146] The determination unit 30 b compares the number of pixels of the edges of the main object area in a plurality of directions DR.

[0147] The determination unit 30 b compares the sums of the lengths of the edges of the main object area in a plurality of directions DR.

[0148] The determining unit 30 b determines the direction DR of relative movement of the imaging unit 20 with respect to the scene according to the comparison result of step S20 (ie, step S21 ).

[0149] The determination unit 30b specifies a direction DR in which the number of pixels of the edge of the entire frame image is the largest among the multiple directions DR, and determines a direction intersecting (e.g., orthogonal to) the specified direction DR as the direction DR in which the imaging unit 20 moves. The determination unit 30b supplies the determination result to the integration unit 30d.

[0150] The determination unit 30b specifies a direction DR in which the sum of the lengths of the edges of the entire frame image is the longest among the multiple directions DR, and determines a direction intersecting (e.g., orthogonal to) the specified direction DR as the direction DR in which the imaging unit 20D moves. The determination unit 30b provides the determination result to the integration unit 30d.

[0151] The determination unit 30b specifies a direction DR in which the number of pixels of the edge of the main object area is the largest among the multiple directions DR, and determines a direction intersecting (e.g., orthogonal to) the specified direction DR as the movement direction DR of the imaging unit 20D. The determination unit 30b supplies the determination result to the integration unit 30d.

[0152] The determination unit 30b specifies a direction DR in which the sum of the lengths of the edges of the main object area is the longest among the multiple directions DR, and determines a direction intersecting (e.g., orthogonal to) the specified direction DR as the direction DR in which the imaging unit 20D moves. The determination unit 30b supplies the determination result to the integration unit 30d.

[0153] The integration unit 30 d integrates the determination results of step S21 (ie, step S22 ) to obtain the direction in which the imaging unit 20 moves.

[0154] The integrating unit 30d may obtain all directions of the one or more directions determined by the determining unit 30b as the direction in which the imaging unit 20 moves. For example, when two directions DR1 / DR3 (see Figure 7 (b)), the integration unit 30d obtains two directions DR1 / DR3 as the directions in which the imaging unit 20 moves.

[0155] Alternatively, the integration unit 30d may prioritize one or more directions determined by the determination unit 30b according to a predetermined rule, and obtain the direction with the highest priority as the direction in which the imaging unit 20 moves. For example, a root having a high angular proximity to the X direction or the Y direction is predetermined, and two directions DR1 / DR3 are determined in step S21 (see Figure 7 In the case of (b), the integration unit 30d prioritizes the two directions DR1 and DR3 in this order. The integration unit 30d obtains the direction DR1 with the higher priority among the two directions DR1 and DR3 as the direction in which the imaging unit 20 moves.

[0156] Alternatively, the integration unit 30d may evaluate one or more directions determined by the determination unit 30b while weighting a plurality of criteria, and obtain a direction having the largest evaluation value as the direction in which the imaging unit 20 moves. For example, regarding the weighting of a plurality of criteria, it is assumed that (the number of pixels of the edge of the entire frame image): (the sum of the lengths of the edges of the entire frame image): (the number of pixels of the edge of the main object area): (the sum of the lengths of the edges of the main object area) = 1:1:2:2. When the direction DR1, the direction DR3, the direction DR2, and the direction DR3 are determined for each of the number of pixels of the edge of the entire frame image, the sum of the lengths of the edges of the entire frame image, the number of pixels of the edge of the main object area, and the sum of the lengths of the edges of the main object area in step S21 (see Figure 7 (b)), the integrating unit 30 d obtains the direction DR3 in which the evaluation value is the largest as the direction in which the imaging unit 20 moves.

[0157] Alternatively, the integration unit 30d may average one or more directions determined by the determination unit 30b and obtain the average direction as the direction in which the imaging unit 20 moves. For example, when two directions DR1 / DR3 (see Figure 7 (c)), the integrating unit 30d obtains a direction DR5 obtained by averaging the two directions DR1 / DR3 as the direction in which the imaging unit 20 moves.

[0158] It should be noted that the number of processes executed in parallel between step S2 and step S19 may be 3 or less, or 5 or more. Figure 15 In this case, the first direction determined in step S21 as a direction intersecting the designated direction in response to the extraction in step S13 and the comparison in step S20 and the second direction determined in step S21 as a direction intersecting the designated direction in response to the extraction in step S14 and the comparison in step S20 may be integrated to obtain a direction as the direction DR in which the imaging unit 20 moves.

[0159] Therefore, the direction in which the imaging unit 20 moves can be multi-dimensionally determined based on a plurality of criteria, and the determination accuracy can be improved.

[0160] In addition, as a seventh modification of this embodiment, the image processing device 1 can enable the learning library to learn multiple scenes and the direction in which the imaging unit 20 moves at that time, and when an event of a predetermined scene is acquired, similar scenes can be searched in the learning library and the direction in which the similar scenes move can be presented.

[0161] In this case, the processing unit 30 stores the learning library LL in the memory 50 (see Figure 3 ). For the learning library LL, a plurality of scenes and the directions in which the imaging unit 20 moves at that time are learned in advance.

[0162] The learning library LL comprises a neural network comprising one or more intermediate layers between an input layer and an output layer. The learning library LL is pre-learned using frame images of a predetermined scene, and the weights of each node in the input layer, one or more intermediate layers, and the output layer are adjusted to appropriately output the movement direction. The learning library LL with the adjusted weights is stored in the memory 50.

[0163] The learning of the learning library LL can be done as follows Figure 16 Execute as shown. Figure 16 1 is a flowchart illustrating the operation of the processing unit 30 in the seventh modification example of the present embodiment.

[0164] In the image processing apparatus 1 , a frame image of a scene to be learned is acquired by the imaging unit 10 (ie, step S31 ), and learning into the library is performed (ie, step S32 ).

[0165] In step S32, the processes of steps S33 to S38 are performed.

[0166] The edge detection unit 30a performs the following steps as shown in step S2 (see Figure 8 ) is performed as in step S3, where edges are detected from the entire region of the frame image (ie, step S33), and information about edges in each direction is extracted as in step S3 (ie, step S34).

[0167] When edge information (e.g., the number of pixels or the sum of lengths) in the selected direction DR is obtained, the edge detection unit 30a confirms whether there is an unselected direction DR among the multiple directions in which edges are to be extracted. The edge detection unit 30a repeats step S34 until all directions in the multiple directions DR are processed (i.e., "No" in step S35).

[0168] When the edge detection unit 30a processes all of the multiple directions (i.e., "Yes" in step S35), the determination unit 30b compares the information of the edge detection results in all directions as in step S5 (step S36). As in step S6, the determination unit 30b determines the direction DR in which the imaging unit 20 moves relative to the scene based on the comparison result in step S36 (i.e., step S37).

[0169] A library is prepared. The library has a neural network including one or more intermediate layers between an input layer and an output layer. Processing unit 30 inputs the frame image acquired in step S31 to the input layer of the library and adjusts the weights of each node in the input layer, one or more intermediate layers, and the output layer so that the output layer of the library outputs a determination result identical to the direction DR determined in step S37 (i.e., step S38).

[0170] When learning (i.e., step S32) is completed, it is determined whether to make the library learn frame images of other scenes (i.e., step S39). In the case of learning frame images of another scene (i.e., "Yes" in step S39), the process returns to step S31, and steps S31 and S32 are performed on the other scene.

[0171] When the library has not learned the frame image of another scene (ie, "No" in step S39), the library is stored in the memory 50 as the learning library LL (ie, step S40).

[0172] The learning library LL can be applied to Figure 17 Use case shown. Figure 17 is a diagram illustrating use cases of the learning library LL in the seventh modification example of the present embodiment.

[0173] For example, when the imaging unit 10 obtains Figure 17 When the frame image IM2 shown in (a) is processed, the processing unit 30 reads the learning library LL from the memory 50 and inputs the frame image IM2 into the input layer LL of the learning library, as shown in FIG. Figure 17 (b) The processing unit 30 extracts the determination result of the moving direction DR1 from the output layer of the learning library LL and provides the determination result to the display control unit 30e (see Figure 5 ). The display control unit 30e superimposes the display object DO1a on the frame image IM2, as shown in FIG. Figure 17 (c) and display is performed on the display screen 41 of the display unit 40 according to the determination result of the direction DR1. The display object DO1a has an arrow shape indicating the moving direction DR1.

[0174] Therefore, a user holding the housing 2 of the image processing device 1 and viewing the display screen 41 of the display unit 40 can be prompted to move the housing 2 in the direction DR indicated by the display object DO1a. In response, when the user moves the housing 2 in the direction DR, the imaging unit 20 moves in the direction DR relative to the scene, thereby effectively increasing the number of events captured by the imaging unit 20. Therefore, in the image processing device 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0175] Alternatively, the learning library LL can be applied to Figure 18 Use case shown. Figure 18 is a diagram illustrating another use case of the learning library LL in the seventh modification example of the present embodiment.

[0176] For example, when the imaging unit 10 obtains Figure 18 When the frame image IM2 shown in (a) is processed, the processing unit 30 reads the learning library LL from the memory 50 and inputs the frame image IM2 into the input layer LL of the learning library, as shown in FIG. Figure 18 (b) The processing unit 30 extracts the determination result of the moving direction DR1 from the output layer of the learning library LL and provides the determination result to the moving control unit 30f (see Figure 5 ). The movement control unit 30f controls the actuator 16 / 26 in the direction DR1 according to the determination result. Figure 18 The lens 11b / 21b of the optical system 11 / 21 is moved in the direction DR1 indicated by the dashed arrow in (c).

[0177] Therefore, since the lens 21b of the optical system 21 moves in a certain direction DR relative to the scene, the number of events captured by the imaging unit 20 can be effectively increased. Therefore, in the image processing device 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0178] Alternatively, the movement control unit 30f controls the actuator 16 / 26 to move in the following manner according to the result of determining the direction DR1. Figure 18 The imaging sensor 12 / 22 is moved in the direction DR1 indicated by the dot-dash arrow in (c).

[0179] Therefore, since the imaging sensor 22 moves in a certain direction DR relative to the scene, the amount of events acquired by the imaging unit 20 can be effectively increased. Therefore, in the image processing device 1, the accuracy of predetermined processing such as object recognition and deblurring using events can be improved.

[0180] Although certain embodiments of the present invention have been described, these embodiments are provided by way of example only and are not intended to limit the scope of the present invention. These novel embodiments may be implemented in a variety of forms, and various omissions, substitutions, and modifications may be made without departing from the core of the present invention. These embodiments and their modifications are intended to be included within the scope and spirit of the present invention and to be incorporated into the description of the claims and their equivalents.

Claims

1. An image processing device, characterized in that include: an image acquisition unit configured to acquire an image of a scene; an event acquisition unit, configured to acquire an event of the scene; a detection unit configured to detect edges in multiple directions from at least a portion of the acquired image; A determining unit is configured to determine a direction in which the event acquiring unit moves relative to the scene according to a detection result of the detecting unit.

2. The image processing device according to claim 1, wherein The determination unit is configured to determine, based on the detection result of the detection unit, a direction in which the event of the scene acquired by the event acquisition unit is appropriately acquired as a movement direction when the event acquisition unit moves relatively.

3. The image processing device according to claim 1, wherein The determining unit is configured to determine, based on the detection result of the detecting unit, a direction in which the event amount of the scene acquired by the event acquiring unit is the largest when the event acquiring unit moves relatively, as a moving direction.

4. The image processing device according to claim 1, wherein The detection unit is configured to detect edges in the entire acquired image in the multiple directions.

5. The image processing device according to claim 4, wherein The determination unit is configured to determine, as the movement direction, a direction intersecting a direction in which the number of pixels of an edge detected in the entire image is the largest among the plurality of directions.

6. The image processing device according to claim 4, wherein The determination unit is configured to determine, as the movement direction, a direction intersecting a direction in which a sum of lengths of edges detected in the entire image is longest, among the plurality of directions.

7. The image processing device according to claim 4, wherein The image processing device further includes: an extraction unit configured to extract a main object area including a main object from the acquired image; The detection unit is configured to detect edges in the extracted main object area in a plurality of directions.

8. The image processing device according to claim 7, wherein The determination unit is configured to determine, as the moving direction, a direction intersecting a direction in which the number of pixels of the edge detected in the extracted main object area is the largest among the plurality of directions.

9. The image processing device according to claim 7, wherein: The determination unit is configured to determine, as the moving direction, a direction intersecting a direction in which a sum of lengths of edges detected in the extracted main object region is longest, among the plurality of directions.

10. The image processing device according to claim 1, wherein The image processing device further includes: The integration unit is configured to integrate the one or more directions determined by the determination unit based on multiple standards to obtain a moving direction.

11. The image processing device according to claim 10, wherein The detection unit is configured to detect edges in the entire acquired image in the multiple directions; The determining unit is configured to determine a first direction and a second direction among the plurality of directions, the first direction intersecting with a direction in which the number of pixels of edges detected in the entire image is the largest, and the second direction intersecting with a direction in which the sum of lengths of edges detected in the entire image is the longest; The integration unit is configured to integrate the first direction and the second direction to obtain a moving direction.

12. The image processing device according to claim 10, wherein: The image processing device further includes: an extraction unit configured to extract a main object area including a main object from the acquired image; The detection unit is configured to detect edges in the extracted main object area in the plurality of directions; The determination unit is configured to determine a third direction and a fourth direction among the multiple directions, the third direction intersecting with a direction in which the number of pixels of the edge detected in the extracted main object area is the largest, and the fourth direction intersecting with a direction in which the sum of the lengths of the edges detected in the main object area is the longest.

13. The image processing device according to claim 10, wherein The integrating unit is configured to obtain all of the one or more directions determined by the determining unit as moving directions.

14. The image processing device according to claim 10, wherein The integration unit is configured to prioritize the one or more directions determined by the determination unit according to a predetermined rule, and obtain the direction with the highest priority as the moving direction.

15. The image processing device according to claim 10, wherein The integration unit is configured to evaluate the one or more directions determined by the determination unit while weighting the plurality of criteria, and obtain a direction with a maximum evaluation value as the moving direction.

16. The image processing device according to claim 10, wherein The integration unit is configured to average the one or more directions determined by the determination unit to obtain an average direction as the moving direction.

17. The image processing device according to claim 1, wherein The image processing device further includes: A display control unit is configured to display the acquired image on the display unit in a display format in which the direction of movement can be identified.

18. The image processing device according to claim 17, wherein The display control unit is further configured to superimpose and display a display object indicating the direction of movement on the acquired image.

19. The image processing device according to claim 1, wherein The event acquisition unit includes an optical system and an actuator, wherein the actuator is configured to move the optical system; The image processing device further includes: A movement control unit is configured to control the actuator to move the optical system in the direction of movement.

20. The image processing device according to claim 1, wherein The event acquisition unit includes an imaging sensor and an actuator, wherein the actuator is configured to move the imaging sensor; The image processing device further includes: A movement control unit is configured to control the actuator to move the imaging sensor in the direction of movement.

Images