Image processing device, image processing method, and program

Abstract

To provide a technique that, in cutting out an image, can prevent an attention object and an object around the attention object from going out of a frame.SOLUTION: An image processing apparatus of the present invention has: image acquisition means that acquires an image including a plurality of objects; detection means that detects the object in the image; setting means that sets an attention object from the plurality of objects detected by the detection means; determination means that determines a first area in the image to include the attention object on the basis of the positions of the objects detected by the detection means; and creation means that creates an image on the basis of the first area determined by the determination means.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an image processing device, a control method, and a program for performing image processing on an image. [Background technology]

[0002] It has been known that a new image is generated by cutting out a part of an image captured by a camera. For example, in Patent Document 1, the direction and position of a face included in a captured image are detected, and a specified face and faces in the direction in which the face is facing are detected as one group to cut out the image. In Patent Document 2, the amount of movement of the area is determined according to the type of scene, and the cut-out position is determined. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6447659 [Patent Document 2] Patent Publication 2018-151979 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not take into consideration faces that are not included in the group, so there is a risk that faces that are not included in the group may be cut off when the image is cut out.

[0005] In addition, in Patent Document 2, the region is set according to the object to be tracked, and the relationship between the set region and the surrounding objects of the tracking target that are not the tracking target is not taken into consideration. Therefore, when cutting out the image, there is a risk that the surrounding objects of the tracking target that are not the tracking target will be cut off.

[0006] Therefore, an object of the present invention is to provide a technique that can prevent an object of interest and objects surrounding the object of interest from being cut off when cutting out an image. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the image processing device of the present invention is characterized in having an image acquisition means for acquiring an image including a plurality of objects, a detection means for detecting objects in the image, a setting means for setting a target object from the plurality of objects detected by the detection means, a determination means for determining a first area in the image so as to include the target object based on the position of the object detected by the detection means, and a generation means for generating an image based on the first area determined by the determination means. Effect of the Invention

[0008] According to the present invention, it is possible to provide a technique that can prevent an object of interest and objects surrounding the object of interest from being cut off when cutting out an image. [Brief description of the drawings]

[0009] [Figure 1] Example of configuration of image processing system 100 according to the first embodiment [Diagram 2] 1A is an example of a hardware configuration of an image processing device 120 according to a first embodiment, and FIG. 1B is an example of a functional configuration of the image processing device 120 according to the first embodiment. [Diagram 3] (a) is an example of an image used for person recognition data, (b) is an example of person recognition data. [Figure 4] Example of setting of divided regions of a person in the first embodiment [Diagram 5] 1 is an example of a flowchart of processing by the image processing device 120 of the first embodiment. [Figure 6] 1 is a flowchart illustrating a process for determining an area to be cut out according to a first embodiment; [Figure 7] 1A is an example of an excision region during the process of determining the excision region in the first embodiment, and FIG. 1B is an example of person recognition data used for determining the excision region in the first embodiment. [Figure 8] An example of an extraction region obtained from the result of human body detection in the image extraction process of the first embodiment. [Figure 9] An example of a cutout area taking into consideration the aspect ratio of the image cutout process of the first embodiment [Figure 10] Example of a user interface of the image processing system 100 according to the first embodiment [Figure 11] Example of configuration of image processing system 1100 according to the second embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the attached drawings. Note that the embodiment described below shows an example of a specific implementation of the present invention, and is one of the specific examples of the configuration described in the claims. Note that in each drawing, the same reference numerals are used for the same components, and duplicated explanations will be omitted.

[0011] (First embodiment) In this embodiment, the position and object area of ​​an object are detected in an image, and a cutout area including the target object is determined based on the detected object area. Then, the image is cut out based on the result of this determination, and a video including the target object is generated.

[0012] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 1. Fig. 1 shows an example of the configuration of an image processing system 100 according to the first embodiment. As shown in Fig. 1, the image processing system 100 in this embodiment is mainly composed of a network camera 110, an image processing device 120, and a network 130.

[0013] The network camera 110 is a camera capable of transmitting control signals via serial communication and IP communication, or transmitting video via analog / digital signals, and can distribute video to external devices, set various internal parameters, and control the camera. It can also record video data to external memory such as an SD card.

[0014] The image processing device 120 is connected to the network camera 110 and can control the camera and receive the distributed video. It also has an application such as a web browser and can control the camera by displaying the video and setting screen distributed from the network camera 110 on a display unit described later. In addition, it can view the video recorded inside the image processing device 120 and control the video generation performed in the image processing device 120. The image processing device 120 can store the video captured by the network camera 110 and generate a video by performing image cropping processing based on the captured video. It can also obtain the video from the network camera 110 via the network 130 described later and store it in its internal memory. In addition, it can obtain data from an external memory such as an SD card, and can also obtain video data recorded by the network camera 110 from the external memory.

[0015] The network 130 is a network for communication between the network camera 110 and the image processing device 120. The form of communication may be wired or wireless, and any form may be used as long as it allows the transmission of video and control signals, such as serial communication and IP communication.

[0016] <Explanation of the configuration of the image processing device> An example of the hardware configuration of an image processing device according to an embodiment of the present invention will be described with reference to Fig. 2(a). Fig. 2(a) shows an example of the hardware configuration of an image processing device 120 according to a first embodiment. The image processing device 120 has a CPU 201, a ROM 202, a RAM 203, a communication unit 204, a display unit 205, an input unit 206, and an internal bus 207 that connects the above components to each other so that they can communicate with each other.

[0017] A CPU 201 executes a control program stored in a ROM 202 (to be described later) and controls each component of the image processing device 120, thereby controlling the entire device. Also, the CPU 201 can obtain image data from a communication unit 204 (to be described later).

[0018] The ROM 202 is a non-volatile storage device such as a flash memory, a hard disk drive (HDD), a solid state drive (SSD), an SD card, etc. The ROM 202 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0019] The RAM 203 is a volatile high-speed storage device such as a DRAM, into which the OS, various programs, and various data are loaded, and which is also used as a working area for the OS and various programs. It also develops programs and makes them executable by the CPU 201.

[0020] The communication unit 204 communicates with the network camera 110 and acquires moving image data. Assumed interfaces for receiving moving image data include general-purpose input interfaces such as HDMI (registered trademark), SDI, and DisplayPort, and LANs connected to a network. In addition, moving images (output moving images) generated by a generating unit 226 (described later) are output to an external device (not shown). Here, the external device includes, for example, a display, a moving image sending device for TV broadcasting or Internet broadcasting, or a recording device such as a USB or DVD.

[0021] The display unit 205 is configured with a CRT or liquid crystal screen, and can display images acquired by the communication unit 204 and the results of processing by the image processing device 120 using images and characters.

[0022] The input unit 206 is configured with user interfaces such as a keyboard, a mouse, and a touch panel, and can be operated by the user to input various instructions to the image processing device 120. Note that, although these components are integrated into one device in Fig. 1, the configuration is not limited to this. For example, these devices may be separate devices, the display unit 205 and the input unit 206 may be integrated, or the communication unit 204 and the display unit 205 may be integrated.

[0023] Next, an example of the functional configuration of the image processing device 120 will be described with reference to the block diagram of Fig. 2(b). Each functional unit shown in Fig. 2(b) may be implemented as hardware or software (computer program). In the latter case, the computer program is stored in the RAM 203. The CPU 201 loads the computer program into the ROM 202 as necessary and executes it to realize the function of the corresponding functional unit.

[0024] The acquisition unit 221 acquires various instructions input by the user operating an input device, and further acquires moving images from an external device (image acquisition means).

[0025] The detection unit 222 has a trained model created using a machine learning method such as deep learning, and detects objects from the moving image or still image acquired by the acquisition unit 221. The detection unit 222 receives an image as input data, assigns a unique ID to an object identified from the positional relationship between the detected objects in frames, and outputs as output data, a score indicating the attributes of the object such as a person and the object area, linked with the frame number, as person recognition data. FIG. 3 shows the person recognition data to be output. FIG. 3(a) shows an image (whole image) acquired from the acquisition unit 221 as input data, and FIG. 3(b) shows person recognition data indicating the unique ID, attributes, and object area of ​​the detected object. At this time, if at least one coordinate of each detected object matches the frame of the acquired whole image, this object is excluded from the target of the determination process of the cutout area. Here, the frame refers to a line connecting the edges of the image, and is composed of a left line, a right line, a bottom line, and an upper line. For example, in FIG. 3(b), ID5 matches the right line and the bottom line of the whole image frame. In this case, ID5 acquires the object area, but does not include it in the cutout area of ​​the determination unit 224 described later. The object attributes refer to people (male, female), animals, immobile objects, others, etc. The attributes are determined from the patterns of the shape features and behavior features. The object area (second area) is an area that is set (determined) so that the detected object does not protrude outside the object area, and is defined by the coordinates of the upper left, lower left, lower right, and upper right of the detected object. The detected object only needs to be located inside the object area that is formed by connecting the coordinates of the upper left, lower left, lower right, and upper right in that order. In addition, the detection unit 222 can determine the object area by dividing each detected object into parts. For example, in the case of a person, as shown in FIG. 4, information on the position of the object is output by dividing one object into three parts, such as a face area 401, an upper body area 402, and a whole body area 403. The object detection method given here is an example, and any form is acceptable as long as the method can detect the necessary object according to the purpose of shooting and the environment. Furthermore, the method of determining the attributes of an object is not limited to the above, and any method that can determine the attributes may be used, such as determining the attributes by a specific object detection process using edge features.

[0026] The setting unit 223 (setting means) sets a target object from the person recognition data output by the detection unit 222. The target object may be set by a method in which the user selects a detected object from a list of objects displayed on the display unit 205.

[0027] The determination unit 224 sets the cutout region of the image based on the object region of the target object and the cutout region information. Specifically, the cutout region of the image is determined by repeating the determination of the cutout region based on the object region of the target object set by the setting unit 223 and the setting information (cutout region information) related to the cutout region (first region). The cutout region information includes various conditions set in advance. The various conditions of the cutout region information are information related to the number and attributes of objects included in the cutout region, priority order, the object region of the target object included in the cutout region, and the aspect ratio. The various conditions of the cutout region information are stored in the ROM 202 and are referred to according to the processing of the determination unit 224. Only the conditions may be stored in the ROM 202, and whether or not to use them for processing may be selected by the determination unit 224. The cutout region information may be determined by a user inputting it from the input unit 206, or may be determined within the image processing device 120.

[0028] The cutout unit 225 performs cutout processing from the image (whole image) acquired by the acquisition unit 221 based on the cutout region determined by the determination unit 224 .

[0029] A generation unit 226 generates a moving image based on the frame numbers labeled by the setting unit 223, information on object regions in the images, and the images cut out by the cut-out unit 225. Furthermore, the image processing device 120 has an internal bus that connects the above-mentioned components to each other so that they can communicate with each other.

[0030] <Explanation of basic processing of image processing device> Next, a basic processing procedure in the image processing device 120 of the present invention will be described with reference to the flowchart in Fig. 5. In this embodiment, a person is detected as a detection target, but a non-human object such as a car or an animal may be detected as a detection target, or a plurality of detection targets such as a person and an animal may be set as detection targets. In addition, a method of detecting an object may be adopted by using a method such as feature point detection.

[0031] This control flow starts when the acquisition unit 221 acquires an instruction to generate a moving image inputted from the input unit 206 .

[0032] In step S501, the communication unit 204 acquires a moving image to be subjected to image processing. However, any method can be used as long as it can acquire image data. For example, the image data may be recorded in a removable external memory such as an SD card and the data may be transmitted to the communication unit 204.

[0033] In step S502, the detection unit 222 detects objects in each frame of the acquired video, and stores the attributes of the objects, such as people, and scores indicating the object regions as person recognition data in the ROM 202. The user can input and set these detection targets using the input unit 206.

[0034] In step S503, the setting unit 223 determines an object to be focused on from among the objects detected in step S502. The method of determining the object to be focused on may be determined by the user selecting a detected object displayed on the display unit 205 from a list of objects displayed on the display unit 205. In this case, the detection information of each object detected in step S502 is read out and converted as data to be displayed on the display unit 205. Furthermore, based on the person recognition data stored in the ROM 202 in step S502, the detection unit 222 labels the object to be focused on set in each frame of the image data. In addition, if it is a person, person authentication can be performed by providing information such as gender and age or learning data of a specific person in advance, and the object to be focused on may be determined using person authentication. If the object to be focused on set by the setting unit 223 is not detected from the frame, nothing needs to be done in the flow of steps S504 and S505.

[0035] In step S504, the determination unit 224 sets a cutout region of the image based on the object region of the target object and the cutout region information set by the setting unit 223 in step S503. Specifically, the cutout region is determined by using setting information (cutout region information) related to the object region of the target object and the cutout region determined in step S503. The cutout region information includes various conditions set in advance. The various conditions of the cutout region information include the number and attributes of objects included in the cutout region, priority order, the region of the target object, information related to the aspect ratio, the proportion of the face region to the area of ​​the cutout region, and the proportion of the area of ​​the cutout region to the area of ​​the entire image. The cutout region information may be determined by a user inputting it from the display unit 205, or may be determined within the image processing device 120. Details of the method of determining the cutout region will be described later.

[0036] In step S 505 , the cutout unit 225 performs image cutout processing on each frame based on the cutout region set in step S 504 , and stores the result in the ROM 202 .

[0037] In step S506, the generation unit 226 combines the image cut out in step S505 with the multiple frames labeled based on the person recognition data stored in step S502. In other words, this control flow ends with the generation of one video including multiple labeled frames. The frames used when generating a video need only be frames in which the target object is located within the image, and it is not necessary to use all of the frames of the acquired video. For example, if the acquired video has a frame in which the target person is not located within the image, that frame does not need to be included in the video to be generated, or a video consisting of all frames may be generated. When generating a video consisting of all frames, the video can be generated by using frames that have not been cut out.

[0038] <How to set the crop area> Next, a specific procedure for determining an area to be cut out for each frame in the present invention will be described with reference to the flow chart in FIG. 6 and FIGS.

[0039] In this embodiment, the control flow shown in Fig. 6 is the details of the control flow of S503 and S504 shown in Fig. 4 in the image processing device 120. In this embodiment, a case will be described where a cutout region is determined for an input image 701 (whole image) in Fig. 7.

[0040] In step S601, the labels (person recognition data) of the object and target object corresponding to each frame of the video are read from the ROM 102, and the object region within the image of the detected object is obtained.

[0041] In step S602, the cutout region is set so that the object regions of all objects detected from the image 701 (whole image) shown in FIG. 7 are contained in the cutout region. The cutout region is set using the coordinates of the ends of the object regions located at the top, bottom, left and right ends of the whole image based on the person recognition data. As an example, consider the processing when persons A to E are present in the image as shown in FIG. 7(a). The person recognition data in FIG. 7(b) indicates the ID, attribute, flag indicating whether or not the person is a person of interest, and coordinates of the object region of the detected person. The coordinates of each object region indicate the coordinates of the upper left, lower left, lower right, and upper right from the left. In order to set the cutout region so that all people are contained, the positions of the upper, lower, left and right ends are determined. The positions of the upper, lower, left and right ends are determined according to the object region of each object. In the case of FIG. 7, the Y coordinates of the coordinates of each object region are compared, and since the position coordinates of the upper left (a2, b2) and upper right (A2, b2) of person B are located at the top end, the upper end of the cutout region is set to match the position of the upper end of the object region of person B. Similarly, in the case of the lower end, the lowermost positions are the lower left (a3, c3) and lower right (A3, c3) of person C, so the lower end of the cutout region is set to match the lower end of the object region of the entire body of person C. As for the left end, the upper left (a3, b3) and lower left (a3, c3) of person C are located at the leftmost positions, so the left end of the cutout region is set to match the left end of the object region of the entire body of person C. Similarly, as for the right end, the upper right (A2, b2) and lower right (A2, c2) of person B are located at the rightmost positions, so the right end of the cutout region is set to match the right end of the object region of the entire body of person B. Therefore, the cutout region is as shown in cutout region 801 in FIG. 8. Finally, in order to match (maintain) the aspect ratio of the original image, correction is performed as shown in cutout region 901 in FIG. 9, taking into account the cutout region when the vertical and horizontal directions are matched to cutout region 801. Specifically, the cutout region information is obtained from the ROM 202, the aspect ratio of the entire image is referenced, and the selected region is expanded vertically and horizontally to match the aspect ratio in Fig. 9. Therefore, in the case of the input image 701 in Fig. 7, a result like the cutout region 702 is obtained by the process of step S602.In this embodiment, the processes from step S603 onwards are performed, but if the setting information on the cutout region (cutout region information) does not include information on the number of objects, the process may end at step S602.

[0042] In step S603, the number of objects (here, the number of people) included in the cutout region is counted. Furthermore, cutout region information is read from ROM 102, and a determination is made regarding the number of objects (the number of people). If the number of objects (the number of people) in the cutout region is greater than N people, the process proceeds to step S604, and if it is N people or less, the process proceeds to step S606.

[0043] In step S604, the people in the cutout region set in step S602 are excluded from the targets to be included in the cutout region so that the number of people in the cutout region is N or less. In this embodiment, the number of people to be included in the cutout region is set in advance by the user using the input unit 106, but this is one example and may be automatically set in the image processing device 120. In addition, the people to be excluded are excluded in order from the person located farthest from the person of interest. For example, in the state of the setting image 702, if the number of objects (number of people) set in advance is 3, C, which is the farthest from the person of interest D, and B, which is the next farthest after C, are excluded. Note that the method of excluding people shown in this embodiment is one example, and other methods may be used. For example, a means may be used in which the farthest person in a position opposite to the moving direction of the person of interest is excluded based on information on the face direction of the person of interest and the motion vector from the previous frame. In addition, the person to be excluded may be determined by the user setting an object to be preferentially excluded from the detected objects.

[0044] In step S605, the cutout region is set so that the object regions of all people to be contained within the cutout region set in step S604 are contained within the cutout region, and then the aspect ratio is adjusted to match that of the entire image. The method of setting the cutout region is the same as in step S602, and a description thereof will be omitted here. Therefore, by setting a frame in the setting image 702 excluding the people determined in step S604, the processing result becomes as shown in the setting image 703. Note that, in this embodiment, the processing from step S606 onwards is performed, but if the setting information on the cutout region (cutout region information) does not include information on the ratio of the face region to the area of ​​the cutout region, the processing may end at step S605.

[0045] In step S606, the ratio of the area of ​​the face area of ​​the target object to the area of ​​the cutout area determined in step S605 is calculated. Next, cutout area information is read from ROM 102, and a determination is made regarding the ratio of the face area. If it is less than X% (threshold value), the process proceeds to step S607, and if it is X% or more, the process proceeds to step S609.

[0046] In step S607, people in the cut-out region set in step S605 are excluded from targets to be included in the cut-out region. Specifically, people in the cut-out region are excluded from targets to be included in the cut-out region until the area of ​​the face region of the target object exceeds X%, and then the process proceeds to step S608. In this embodiment, the threshold value for the proportion of the face region is set in advance by the user using the input unit 106, but this is only an example and may be automatically set in the image processing device 120.

[0047] In step S608, the cutout region is set so that the object regions of all people to be contained in the cutout region set in step S607 are contained in the cutout region, and then the aspect ratio is adjusted to match that of the entire image. The method of setting the cutout region is the same as in step S602, and a description thereof will be omitted here. Therefore, by setting the cutout region 703 while excluding the people determined in step S607, the processing result is as shown in cutout region 704. Note that, in this embodiment, the processing from step S609 onwards is performed, but if information on the ratio of the area of ​​the cutout region to the area of ​​the entire image is not included, the processing may end in step S608.

[0048] In step S609, the ratio of the area of ​​the cutout region set in step S608 to the area of ​​the entire image is calculated. Next, cutout region information is read from ROM 102, and a judgment is made regarding the ratio calculated in step S609. If it is smaller than P% (threshold), the process proceeds to step S610. In this embodiment, the threshold setting for the ratio of the area of ​​the cutout region to the area of ​​the entire image is set in advance by the user using input unit 106, but this is just an example and may be automatically set within the image processing device. For example, a ratio that does not reduce visibility may be automatically calculated and set by inputting the resolution of the display device that outputs the area of ​​the entire image.

[0049] In step S610, the object that is not included in the cutout region in step S609 is included in the cutout region, and the process proceeds to step S611. The object to be included is the person located closest to the person of interest. For example, in the state of the setting image 704, among C, A, and B that are outside the cutout region, A that is closest to the person of interest D is included in the cutout region. Note that the method of adding a person shown in this embodiment is an example, and other methods may be used. For example, a means may be used in which a person in the same traveling direction as the person of interest is added based on information on the face direction of the person of interest and the motion vector from the previous frame. Also, the person to be added may be determined by the user setting an object to be preferentially added from among the detected objects.

[0050] In step S611, the cutout region is set so that the area of ​​the cutout region determined in step S606 is equal to or greater than the threshold value set in the cutout region information with respect to the area of ​​the entire image. The method of setting the cutout region is the same as in step S602, and the cutout region is set so that the object regions of all people to be contained within the cutout region set in step S610 are contained within the cutout region. Therefore, by adding the people determined in step S610 and setting the cutout region, a processing result like cutout region 705 is obtained.

[0051] In step S612, a frame is determined as the cut-out region, and the process ends. Note that the order, number of times, conditions, and method of the frame resetting procedure in this embodiment are merely examples, and can be modified according to the purpose of implementation.

[0052] With the above-described configuration, this embodiment can provide a technique that can prevent an object of interest and objects surrounding the object of interest from being cut off when an image is cut out.

[0053] Here, the user interface of the image processing device 120 in the image processing system 100 will be described with reference to FIG.

[0054] First, the user accesses the generation unit 226 of the image processing device 120 from the input unit 206. From the user interface, by selecting a menu from button 1001, it is possible to generate a moving image, view recorded moving images, and view the generated moving image. When generating a moving image, first, a moving image to be cut out is selected with button 1002. At this time, the user can select from the moving image data stored in the image processing device 120. It is also possible to upload moving image data via a network.

[0055] Next, a person of interest is selected in item 1003. The person can be selected from the displayed face images, but it is also possible for the user to upload an image using a registration button. The image processing device 120 uses the image selected here as preliminary data for the recognition process. In this embodiment, recognition is performed using a face image of a person as the target, but this is only an example, and the preliminary data used for recognition may change depending on the object of interest. Also, a form in which parameters such as numerical values ​​rather than images are input from a user interface may be used.

[0056] In item 1004, it is possible to select which part of the person of interest is to be included in the cut-out region. The item selected here is used as a parameter for the position of the edge of the object cut-out region when determining the position of the edge of the cut-out region. For example, when the whole body of the person of interest is to be captured, the position of the edge of the cut-out region of the whole body of the object located at the edge is used when determining the position of the edge of the cut-out region. Finally, when the button 1005 is pressed, the image processing device 120 starts the image cut-out process, and after the process is completed, the user can check the generated video from the generated video list screen. The user interface shown in this embodiment is an example, and any form is possible as long as the user can perform operation based on the captured video image to generate a video.

[0057] Second Embodiment In the first embodiment, image processing has been described in the case where the image processing device 120 and the network camera 110 constitute the image processing system 100. Next, image processing will be described in the case where the image processing system 1100 is constituted by the network camera 1101, the information terminal 1102, and the server 1104 as shown in Fig. 11. In the second embodiment, the server device 1104 performs the image processing shown in Figs. 5 and 6.

[0058] The image processing system 1100 in this embodiment is mainly composed of a network camera 1101 , an information terminal 1102 , a network 1103 , and a server device 1104 .

[0059] The network camera 1101 is a camera capable of transmitting control signals via serial communication and IP communication, or transmitting video via analog / digital signals, and can distribute video to external devices, as well as set various internal parameters and control the camera. It can also record video data to external memory such as an SD card.

[0060] The information terminal 1102 can connect to the network camera 1101, control the camera, and receive distributed video. It also has applications such as a web browser, and can display the video and setting screens distributed from the network camera 1101 and control the camera. In addition, it can connect to a server device 1104, and can view the video recorded in the server device 1104 and remotely control the video generation performed within the server device 1104.

[0061] A network 1103 is a network for communication between a network camera 1101, an information terminal 1102, and a server device 1104. The form of communication may be wired or wireless, and any form may be used as long as it allows transmission of video and control signals, such as serial communication and IP communication.

[0062] The server device 1104 can store images captured by the camera, perform image cropping on the captured images, and generate new images. It can also obtain images from the network camera 1101 via the network 1103 and store them in its internal memory. In addition, it can obtain data from an external memory such as an SD card, and can also obtain video data recorded by the network camera 1101 from the external memory.

[0063] Here, the image processing procedure in the image processing system 1100 of this embodiment will be described with reference to the flowchart in Fig. 5. First, in step S501, the server device 1103 acquires the image to be cropped. In this embodiment, the image is acquired by receiving the image data distributed from the network camera 1101 via the network 1103. However, any method can be used as long as it can acquire the image data. For example, the network camera 1101 may record the image data in a removable external memory such as an SD card when shooting, and transfer the data to the server device 1103 using the external memory.

[0064] Next, in step S502, the server device 1103 performs object detection processing for each frame of the video data. The detection targets refer to distinctive objects such as people, background patterns, and signs. The user can input and set these detection targets using the Web browser of the information terminal 1102. In addition, a method of detecting objects using feature point detection or the like may be adopted. Furthermore, the server device 1103 can partially acquire the position of a detected object. In step S503, the server device 1103 performs object determination for the object detected in step S502 in each frame. The server device 1103 labels the object area detected in each frame of the video data, and stores the frame number and the object area in the image in association with the label.

[0065] In step S504, a region to be cut out of the image is determined based on the position of the detected object. The information on the object region stored in step S503 is used to determine the region to be cut out. The details of the method for determining the region to be cut out are the same as those in the first embodiment.

[0066] In step S505, image extraction processing is performed on each frame based on the determined extraction area. Finally, in step S506, all frames are combined to generate one image, and the process ends.

[0067] With the above-described configuration, this embodiment can provide a technique that can prevent an object of interest and objects surrounding the object of interest from being cut off when an image is cut out.

[0068] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0069] 100 Image Processing System 110 Network Camera 120 Image Processing Device 130 Network 1100 Image Processing System 1101 Network Camera 1102 Information terminal 1103 Network 1104 Server

Claims

1. An image acquisition means for acquiring an image, A detection means for detecting multiple objects from the aforementioned image, A setting means for setting an object of interest from the plurality of objects detected by the detection means, A determination means that determines a first region in the image, including the object of interest and objects other than the object of interest, based on the position of the object detected by the detection means. A generation means for generating an image based on the first region, An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, wherein the determination means determines the first region based on one of the following: the number of objects included in the first region, the ratio of the region corresponding to the face of the object of interest to the first region, and the ratio of the first region to the image acquired by the image acquisition means.

3. The image processing apparatus according to claim 1, characterized in that the determination means determines the first region so as to maintain the aspect ratio of the image acquired by the image acquisition means.

4. The image processing apparatus according to claim 1, characterized in that the determination means determines the first region such that the number of objects included in the first region becomes less than the threshold when the number of objects included in the first region is greater than the threshold.

5. The image processing apparatus according to claim 1, characterized in that the determination means determines the first region by reducing the number of objects to be included in the first region when the size of the region corresponding to the face of the object of interest is smaller than a predetermined ratio to the size of the first region.

6. The image processing apparatus according to claim 1, wherein the determination means determines the first region by increasing the number of objects to be included in the first region if the area of ​​the first region is smaller than a predetermined ratio to the area of ​​the image obtained by the image acquisition means.

7. The image processing apparatus according to claim 1, characterized in that the attributes of the object detected by the detection means include people, animals, and stationary objects.

8. The image processing apparatus according to claim 1, characterized in that the generation means generates an image by cutting out the first region from the image.

9. The image processing apparatus according to claim 1, characterized in that the determination means excludes from the objects to be included in the first region by the determination means any region of an object detected by the detection means in which at least a portion of the region of the object coincides with the outer periphery of the image.

10. The image processing apparatus according to claim 1, characterized in that the setting means sets the object of interest by authentication based on reference information entered by the user.

11. The image processing apparatus according to claim 1, characterized in that the determination means determines the first region of the object of interest using regions corresponding to parts specified by the user from among the face, upper body, and whole body.

12. The image processing apparatus according to claim 1, wherein the determination means determines the object to be excluded based on motion information representing the direction of movement of the object to be excluded when excluding an object other than the object of interest to be included in the first region.

13. The image processing apparatus according to claim 1, wherein the determination means determines the object to be excluded based on information representing the orientation of the object of interest when excluding an object other than the object of interest to be included in the first region.

14. The image processing apparatus according to claim 1, wherein the determination means selects objects other than the object of interest to be included in the first region based on a priority set by the user.

15. The image is a moving image, The image processing apparatus according to claim 1, characterized in that the generation means selects a frame in which the object of interest is detected and generates a video using the image based on the first region.

16. The image processing apparatus according to claim 15, characterized in that the generation means includes frames in which the object of interest is not detected in the video, in which no cutout is performed.

17. The image processing apparatus according to claim 1, characterized in that the determination means automatically sets a threshold value for the area of ​​the first region based on the resolution of the output destination display device.

18. An image processing method performed by one or more processors, The image acquisition process involves obtaining an image, A detection step for detecting multiple objects from the aforementioned image, A setting step in which a target object is selected from the plurality of objects detected in the detection step, A determination step in which, based on the position of the object detected in the detection step, a first region in the image is determined to include the object of interest and objects other than the object of interest, A generation step of generating an image based on the first region, An image processing method characterized by having the following features.

19. An image acquisition step of acquiring an image, A detection step for detecting multiple objects from the aforementioned image, A setting step in which a target object is selected from the plurality of objects detected in the detection step, A determination step in which, based on the position of the object detected in the detection step, a first region in the image is determined to include the object of interest and objects other than the object of interest, A generation step of generating an image based on the first region, A program for causing a computer to execute an image processing method that includes the following.

Images