Data collection system and data collection method for additional learning

Abstract

To provide a technique to efficiently collect data for additional learning.SOLUTION: A data collection system for additional learning of the present invention comprises: an image processing function unit that performs image processing of recognizing and tracking a target object included in images obtained by photographing a predetermined area; and a storage processing function unit that stores the data for additional learning of the image processing. The storage processing function unit temporarily stores data, calculates a predetermined index from the temporarily stored data, performs priority calculation processing based on the index, and performs storage determination processing on the temporarily stored data. When a data amount in an additional learning data unit is less than a threshold, the storage determination processing stores the temporarily stored data in the additional learning data unit, and when the data amount in the additional learning data unit is equal to or more than the threshold, compares the temporarily stored data with data with the lowest priority of the data stored in the additional learning data unit through the priority calculation processing, and deletes the data with the lower priority.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a data collection system and method for incremental learning. [Background technology]

[0002] Conventionally, surveillance systems that track target objects have been used in various situations in public facilities and local governments, such as tracking passengers moving on train platforms. Such surveillance systems require image processing technology to recognize target objects in images captured by cameras and track their movement.

[0003] For example, Patent Document 1 discloses an intrusion detection system that has a control unit that detects an image of an object in a video signal provided by an imaging unit so that the object on the screen can be reliably tracked while being centered, and moves the image of the object in the pan and tilt directions using a first drive unit and a second drive unit so that the image of the object is positioned in the center of the video screen in accordance with the movement and changes of the object, and further controls the third drive unit to zoom in or out so that the image of the object maintains a predetermined size. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5514506 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, image processing technology that uses AI to recognize objects in images has been applied to surveillance systems such as those shown in Patent Document 1. If a malfunction occurs during operation of a system using such image processing technology, such as the overlooking of a specific image pattern, image processing performance can be improved by collecting additional learning data that includes the specific image pattern and performing additional learning.

[0006] However, the conventional method of collecting data for additional learning required workers to manually extract the necessary scenes from footage around the time of an oversight or other malfunction, and collect the data for additional learning, which required a great deal of work.In addition to the method performed by workers, there is also a method of constantly distributing and storing data, but since unnecessary footage is also distributed and stored, there is a problem in that it puts pressure on data storage capacity and communication lines, increasing storage and communication costs.

[0007] Therefore, an object of the present invention is to provide a technique for efficiently collecting data for additional learning. [Means for solving the problem]

[0008] To solve the above-mentioned problems, one representative data collection system for additional learning of the present invention is a data collection system for additional learning that includes an image processing function unit that performs image processing to recognize and track target objects included in video captured of a predetermined area, and a storage processing function unit that stores data for additional learning of the image processing. The storage processing function unit temporarily stores data, calculates a predetermined index from the temporarily stored data, performs a priority calculation process based on the index, and performs a storage determination process for the temporarily stored data. The storage determination process stores the temporarily stored data in the additional learning data unit if the amount of data in the additional learning data unit is less than a threshold. Furthermore, if the amount of data in the additional learning data unit is equal to or greater than the threshold, the priority calculation process compares the temporarily stored data with the data with the lowest priority (hereinafter referred to as "lowest-priority data") among the data stored in the additional learning data unit, and deletes the data with the lower priority. [Effects of the Invention]

[0009] According to the present invention, data for additional learning can be collected efficiently. Problems, configurations, and effects other than those described above will become apparent from the following description of the preferred embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a data collection system for additional learning according to an embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing a first example of an image to be subjected to image processing. [Figure 3] FIG. 3 is a schematic diagram showing a second example of an image to be subjected to image processing. [Figure 4A] FIG. 4A is a flowchart showing image processing. [Figure 4B] FIG. 4B is a flowchart of the image processing. [Figure 5] FIG. 5 is a schematic diagram showing image processing information associated with the processed video signal Svi. [Figure 6] FIG. 6 is a diagram schematically showing a list of data stored in the additional learning data section. [Figure 7] FIG. 7 is a flowchart of the accumulation process. [Figure 8] FIG. 8 is a flowchart of the accumulation process of the first modification. [Figure 9] FIG. 9 is a flowchart showing the image processing of the second modification. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0012] <System configuration> The configuration of a data collection system for additional learning according to an embodiment of the present invention will be described with reference to FIG. 1 is a diagram showing a data collection system for additional learning according to an embodiment of the present invention. The data collection system 1 includes a camera unit 10, an image processing server 20, and a storage server 30. Video signals are communicated between the components of the data collection system, and the communication means and video communication standard can be selected appropriately depending on the situation in which the present invention is applied.

[0013] The camera unit 10 captures an image of a predetermined area and generates a video signal Sv that includes at least information about the date and time of capture. The camera unit 10 transmits the generated video signal Sv to an image processing server 20, which will be described later. Note that although the number of camera units 10 shown in FIG. 1 is one, the number of camera units is not limited to this. A configuration including multiple camera units is also possible.

[0014] The image processing server 20 includes an input unit 21, a recognition processing unit 22, an output unit 23, a learning unit 24, and a learning data unit 25. However, the learning unit 24 and the learning data unit 25 may be provided separately from the image processing server 20.

[0015] The input unit 21 receives the video signal Sv generated by the camera unit 10. The input unit 21 may or may not be present. While Fig. 1 shows the video signal Sv being transmitted from the camera unit 10, a configuration in which a storage device in which video data is accumulated is connected to the image processing server 20 may also be used, and the method of transferring data is not limited.

[0016] The recognition processing unit 22 performs image processing to recognize and track a predetermined target object from the video signal Sv. Here, image recognition is performed by AI (artificial intelligence) using a trained model that has been trained to recognize the predetermined target object. The learning unit 24 learns using training data stored in the training data unit 25, and creates a trained model that recognizes the predetermined target object. Thereafter, additional training data is supplied to the training data unit 25. The trained model is then updated in the learning unit 24 using the additional training data.

[0017] Furthermore, the recognition processing unit 22 generates recognition information indicating either "recognized" or "not recognized (unrecognized)." "Not recognized (unrecognized)" includes cases where the target object is not recognized because it moves out of the camera frame, as well as cases where it is not recognized due to a malfunction such as an oversight.

[0018] The recognition processing unit 22 also generates number information indicating the number of objects recognized in the camera frame that should be tracked. The recognition processing unit 22 also generates tracking start information when a predetermined target object is recognized and tracking begins, and tracking end information when tracking ends without the target object being recognized. The recognition processing unit 22 associates the above-mentioned recognition information, number information, tracking start information, and tracking end information with the video signal Sv as image processing information. The output unit 23 outputs a processed video signal Svi in ​​which the image processing information is associated with the video signal Sv to the storage server 30. The output unit 23 may or may not be present, and the method of transferring data is not limited.

[0019] The storage server 30 includes an input unit 31, a temporary storage unit 32, a determination unit 33, and an additional learning data unit 34. The storage server 30 determines whether the processed video signal Svi includes a portion that can be used as additional learning data, and collects the data.

[0020] The input unit 31 receives the processed video signal Svi. The input unit 31 may or may not be present, and the method of transferring data is not limited. The temporary storage unit 32 extracts a portion of the processed image signal Svi that will serve as data for additional learning, and temporarily stores it as temporarily stored data. The determination unit 33 determines the priority based on image processing information associated with the portion of the video stored in the temporary storage unit, and determines whether to store the temporarily stored data in the additional learning data unit 34. The additional learning data unit 34 saves data for additional learning in the learning data unit 25, and provides learning data to the learning data unit 25 as necessary.

[0021] <Image processing> The image processing performed in the image processing server 20 will be described with reference to FIGS. In this embodiment, a case will be described in which the data collection system 1 is applied to a surveillance system that recognizes and tracks people who enter a predetermined area. Here, the camera unit 10 is a fixed camera that captures images of a predetermined area. However, the present invention is not limited to surveillance systems.

[0022] FIG. 2 is a schematic diagram showing a first example of an image to be subjected to image processing.

[0023] FIG. 2(a) shows a scene in which the recognition processing unit 22 recognizes person A and begins tracking them. Here, the recognition processing unit 22 acquires and processes images of a predetermined area frame by frame. Frame 40 represents one frame contained in the video signal Sv at a certain point in time. The frame rate is, for example, 30 fps (frames per second). The recognition processing unit 22 recognizes that person A is included in frame 40. The recognition processing unit 22 recognizes and begins tracking person A from frame 40. In addition to person A, the recognition processing unit 22 recognizes five other people 44a to 44e in frame 40 and generates number information (6 [people]). In FIG. 2(a), objects recognized as targets for image processing are shown surrounded by dashed lines. Object 43 is not recognized as a target for image processing.

[0024] In a frame at any point in time, a central region Rc is set, which is a region within a radius r1 from a point near the center of the frame, an intermediate region Rm that is concentric with the central region and is between the radius r1 and a radius r2 that is larger than the radius r1, and an edge region Re that is a region farther away from the intermediate region Rm. Setting such regions is an example of a means for identifying the position of the tracked object, and is not limited to this and may not be required.

[0025] FIG. 2(b) is a diagram showing frame 41, which is taken several frames after frame 40 in FIG. 2(a). In frame 41, person A moves from the upper left to the lower right in the photographed area. The recognition processing unit 22 tracks the movement of person A in this manner. Furthermore, the recognition processing unit 22 recognizes four other people 45a to 45d in addition to person A in frame 41 and generates number information (5 [people]).

[0026] 2(c) is a diagram showing frame 42, which is the next frame after frame 41 in FIG. 2(b). In frame 42, person A is not recognized. If the determination condition for determining tracking of person A is met, the recognition processing unit 22 ends tracking of person A.

[0027] Here, the determination criterion for ending tracking is the time during which person A remains unrecognized from frame 42. This time can be set according to the location of person A in the frame before the frame in which person A became unrecognized. For example, if person A was in the center region Rc in the frame immediately before the frame in which person A became unrecognized, tracking ends when person A remains unrecognized for 10 seconds. If person A was in the middle region Rm in the frame immediately before the frame in which person A became unrecognized, tracking ends when person A remains unrecognized for 5 seconds. If person A was in the edge region Re in the frame immediately before the frame in which person A became unrecognized, tracking ends when person A remains unrecognized for 1 second.

[0028] Such a determination condition for ending tracking can be set appropriately depending on the situation to which this embodiment is applied. For example, when the camera unit 10 captures a wide area such as a station platform, duration values ​​corresponding to the central region Rc, the middle region Rm, and the edge region Re are set as described above, while when the camera unit 10 captures a narrow area such as a station ticket gate, a uniform duration of 1 second can be set without dividing the areas. The determination condition for ending tracking can be set depending on the situation, such as the location where the surveillance system is used and the expected flow of people.

[0029] Fig. 3 is a schematic diagram showing a second example of an image to be subjected to image processing. Fig. 3 differs from the first example shown in Fig. 2 in that a malfunction has occurred and a person has not been recognized. Components that are the same as or equivalent to those shown in Fig. 2 are given the same reference numerals, and their description will be simplified or omitted.

[0030] FIG. 3(a) is a diagram showing a scene in which the recognition processing unit 22 recognizes person B and begins tracking them. The recognition processing unit 22 recognizes that person B is included in frame 50. The recognition processing unit 22 recognizes and begins tracking person B from the time of frame 50. Furthermore, the recognition processing unit 22 recognizes five other people 54a to 54e in addition to person B in frame 50 and generates number information (6 [people]). In FIG. 3(a), objects recognized as targets for image processing are displayed as surrounded by dashed lines. Object 43 is not recognized as a target for image processing.

[0031] FIG. 3(b) is a diagram showing frame 51, which is taken several frames after frame 50 in FIG. 3(a). In frame 51, person B moves from the upper left to near the center of the photographed area. The recognition processing unit 22 tracks the movement of person B in this manner. In addition to person B, the recognition processing unit 22 also recognizes four other people 55a to 55d in frame 51 and generates number information (5 [people]).

[0032] FIG. 3(c) is a diagram showing frame 52, which follows frame 51 in FIG. 3(b). In frame 52, a person corresponding to person B is actually included, but the recognition processing unit 22 does not recognize person B. Because person B was in the intermediate region Rm in frame 51, the recognition processing unit 22 sets the determination condition as whether person B remains unrecognized for five seconds. If the determination condition is met (person B has not been recognized for five seconds), the recognition processing unit 22 ends tracking of person B.

[0033] 2 and 3, the recognition and tracking of only one specific person has been described, but the present embodiment is not limited to this. It is also possible to distinguish and recognize and track multiple people.

[0034] 4A and 4B are diagrams showing a flowchart of image processing.

[0035] 4A, the image processing server 20 receives the video signal Sv from the camera unit 10. The video signal Sv is received via the input unit 21.

[0036] In step S2, the image processing server 20 performs recognition processing on a predetermined target object using AI (artificial intelligence) that uses a trained model for the video signal Sv.

[0037] Tracking process 1, which begins in step S3, is related to object recognition and tracking. In step S3, the recognition processing unit 22 determines whether a predetermined target object has been recognized in the video. Here, object recognition can be performed, for example, on a frame-by-frame basis for the video signal. In step S3, the recognition processing unit 22 also associates, with the frame, recognition information indicating whether the predetermined target object has been "recognized" or "not recognized (unrecognized)," information about the predetermined target object (position information and feature values), and number information about the predetermined target object. The association of number information is continued throughout the image processing shown in FIG. 4. That is, in the case of FIG. 2(a) as an example, the information associated with the frame in step S3 is "recognized," the position information and feature values ​​of person A and persons 44a to 44e, and the number information "6."

[0038] In step S3, the recognition processing unit 22 performs a tracking process on the recognized predetermined target objects. The tracking process is performed on each of the recognized predetermined target objects, and it is determined whether or not there is associated tracking information.

[0039] In step S4, the recognition processing unit 22 determines whether the recognized predetermined target object is associated with the object being tracked. If the recognized predetermined target object is not associated with the object being tracked (YES in step S4), the recognition processing unit 22 adds tracking information as a newly appeared object and adds tracking start information in step S5. If the recognized predetermined target object is associated with the object being tracked (NO in step S4), the recognition processing unit 22 updates the tracking information in step S6. After executing step S5 or step S6, the process proceeds to step S7.

[0040] In step S7, the recognition processing unit 22 determines whether all predetermined target objects recognized in step S2 have been processed. If the determination condition is not met (NO in step S7), the process returns to tracking process 1 in step S3 and continues. If the determination condition is met (YES in step S7), the process proceeds to step 8 in FIG. 4B.

[0041] In step S8, the tracking process 2 starts. First, the recognition processing unit 22 extracts one piece of tracking information.

[0042] In step S9, the recognition processing unit 22 determines whether the tracking start time is not the current time (the tracking information is not added in step S5) and whether the tracking information has not been updated (the information has not been updated in step S6). If the determination condition is met (YES in step S9), the recognition processing unit 22 associates information indicating unrecognized information and proceeds to step S10. If the determination condition is not met (NO in step S9), the tracking is in progress and the processing proceeds to step S13.

[0043] In step S10, the recognition processing unit 22 sets a determination condition for ending the tracking process based on the position of a predetermined target object in the frame immediately before the frame in which the target object has become unrecognized.

[0044] In step S11, the recognition processing unit 22 determines whether the determination condition is met. If the determination condition is not met (NO in step S11), the process proceeds to step S13. If the determination condition is met (YES in step S11), the recognition processing unit 22 ends tracking of the predetermined target object and adds tracking end information indicating that tracking has ended (step S12).

[0045] In step S13, if tracking end information has not been added to the tracking information associated with the predetermined target object (NO in step S13), the recognition processing unit 22 repeats tracking process 2. If tracking end information has been added to the tracking information associated with the predetermined target object (YES in step S13), the recognition processing unit 22 proceeds to step S14 in FIG.

[0046] Then, in step S14 of Fig. 4A, the recognition processing unit 22 determines whether image processing has been completed up to the end of the image included in the video signal Sv. If there is remaining image (NO in step S14), the recognition and tracking processing of the predetermined target object continues. If image processing has been completed up to the end of the image (YES in step S14), the image processing ends.

[0047] The above-described image processing is performed continuously during the operation of the surveillance system to which the data collection system 1 is applied. In this way, data for additional learning can be efficiently collected from the vast amount of video data generated during the operation of the surveillance system.

[0048] <Storage processing> 5 to 7, the temporary storage process and index calculation performed by the temporary storage unit 32 of the storage server 30, and the storage determination process and priority calculation process performed by the determination unit 33 will be described. Note that the index calculation may be performed by the determination unit 33.

[0049] (Temporary storage processing) FIG. 5 is a schematic diagram showing image processing information associated with the processed video signal Svi. FIG. 5 schematically shows the case where steps S3 in FIG. 4A to S12 in FIG. 4B have been performed. FIGS. 5(a) to 5(c) show an example where the video signal contains three predetermined target objects. In FIG. 5, the video signal is divided into frames along the time axis (t). Frames associated with recognition information indicating that the predetermined target objects have been recognized are hatched. Frames to which tracking start information has been added are marked with the letters ps1 and ps2 and a triangle symbol, and frames to which tracking end information has been added are marked with the letters pf1 and pf2 and a triangle symbol. FIG. 5(a) shows that after tracking of the predetermined target object 1 has been temporarily terminated at pf1, the tracking information has been updated at ps2, and tracking has resumed. The tracking end determination condition is met at pf2. In Fig. 5(b), tracking of a predetermined target object 2 is started at psn, and tracking continues after the frame shown in Fig. 5. In Fig. 5(c), tracking of a predetermined target object 3 is started before the frame shown in Fig. 5, and is temporarily completed at pfm. In Fig. 5(c), tracking information is updated, and tracking is started again at psm+1, and is completed at pfm+1.

[0050] The processed image signal Svi is transmitted to the input unit 31 of the storage server 30 and then temporarily stored in the temporary storage unit 32. The temporary storage unit 32 extracts frames from the video data included in the processed image signal Svi, from the frame to which tracking start information is added to the frame to which tracking end information is added, and temporarily stores the extracted frames as temporarily stored data. In the example of FIG. 5, a frame group vf1 including frames from the tracking start information ps1 to the tracking end information pf1 is designated as the temporarily stored data. The temporary storage unit 32 also associates with the frame group vf1 a tracking start time ts1 indicating the time when tracking corresponding to the tracking start information ps1 started and a tracking end time tf1 indicating the time when tracking corresponding to the tracking end information pf1 ended. Frame groups vf2, vfn, vfm, and vfm+1 are similarly temporarily stored. The following description will be given using the frame group vf1 as an example.

[0051] (Calculation of indicators) The temporary storage unit 32 calculates indices from the image processing information associated with the frame group vf1. In this embodiment, the calculated indices include the tracking time, the number of unrecognized events, the number of objects in the area (maximum value), and the video start date and time. However, the indices are not limited to these.

[0052] The tracking time is calculated by finding the difference between the tracking start time ts1 and the tracking end time tf1. For example, in the case of frame group vf1, there are 15 frames between the tracking start time ts1 and the tracking end time tf1, so if the frame rate is 30 fps, the tracking time is 15 / 30 = 0.5 seconds. Note that the index is not limited to the tracking time. For example, the number of frames can also be used.

[0053] The number of unrecognized instances is the number of times a predetermined target object has not been recognized in the frame group vf1. For example, one method is to count one unrecognized frame as one unrecognized instance. In FIG. 5(a), there are nine unrecognized frames between the frame to which the tracking start information ps1 is added and the frame to which the tracking end information pf1 is added, so the number of unrecognized instances is counted as nine. Note that the method for evaluating the unrecognized state is not limited to this. For example, instead of the number of unrecognized instances, the duration of the recognized state can also be used as an index for evaluating the unrecognized state.

[0054] The number of objects (maximum value) in an area (hereinafter also referred to as the "maximum number of objects") refers to the maximum value of the numerical information of each frame included in the frame group vf1. Note that the value used as an index is not limited to the maximum value. For example, the average value or median value of the numerical information can also be used as an index.

[0055] The video start date and time is the date and time when the generation of the video signal Sv including the frame group vf1 starts, although other dates and times may also be used as the index.

[0056] (Accumulation determination process) The determination unit 33 determines whether or not to store the temporarily stored data in the additional learning data unit 34. To do this, the determination unit 33 first determines whether or not the amount of data stored in the additional learning data unit 34 has reached a threshold. If the threshold has not been reached (if it is less than the threshold), the determination unit 33 stores the temporarily stored data (frame group vf1) in the additional learning data unit 34. At this time, the data list (described later) is updated according to new priorities determined in a priority calculation process described later.

[0057] If the threshold value has been reached (if the threshold value is reached or exceeded), the judgment unit 33 judges which data to accumulate based on the priority determined by the priority calculation process, between the data accumulated in the additional learning data unit 34 and the temporarily accumulated data (frame group vf1). Note that the threshold for the amount of data may be, for example, the upper limit of the storage capacity of the additional learning data unit 34, or a lower limit of the free space remaining for accumulating the next data. Alternatively, the upper limit of the number of data items (number of cases) accumulated in a predetermined period may be used as the threshold, such as limiting the number of data items to 10 per day.

[0058] FIG. 6 is a diagram schematically illustrating a list of data (hereinafter referred to as a "data list") stored in the additional learning data unit 34. In determining priority and storage in this embodiment, "data" is handled in units of frame groups (such as frame group vf1). The item "priority" is a value calculated by the priority calculation process in the determination unit 33 based on the index of the stored frame group. The index of the data stored in the additional learning data unit 34 is the same as the above-mentioned index calculated for the temporarily stored data, so a description thereof will be omitted. Note that, in addition to the items shown in FIG. 6, it is also possible to add items to the data list as appropriate. For example, an index indicating the type of a predetermined target object may be added. Such added indexes can also be taken into consideration in the priority calculation process described below.

[0059] For example, in FIG. 6, the frame group with the highest priority has a tracking time of 10 minutes, 20 unrecognized times, and 10 targets (maximum) in the area. The data list also includes data with the lowest priority (hereinafter referred to as "lowest data"). In FIG. 6, the priority R L The frame group of R corresponds to the lowest data. L The frame group has a tracking time of T L minutes, the number of unrecognized times is C L The number of targets in the area (maximum value) is N times. LAll of the data listed in the data list in Figure 6 are frames extracted from the same video signal Sv, so they have the same video start date and time, but it goes without saying that the video start dates of data extracted from different video signals may be different.

[0060] The judgment unit 33 judges whether or not to accumulate the temporarily accumulated data in the additional learning data unit 34 based on the priority of the index. If it is determined that the data cannot be accumulated, the temporarily accumulated data is deleted as is. If it is determined that the data can be accumulated, the lowest-ranked data in the additional learning data unit 34 is deleted. Then, the data list is updated according to the new priority determined in the priority calculation process.

[0061] After the temporary storage data has been stored or deleted in this way, if there is next temporary storage data, the same storage determination process is repeated.

[0062] (Priority calculation process) The method of assigning priorities based on the index (priority calculation process) will be described. First, the tracking time is compared. The determination unit 33 determines whether the tracking time of the temporarily stored data is equal to or greater than the lowest data (R L ) is the tracking time of the temporary storage data. L ) tracking time (T L ), the determination unit 33 deletes the temporarily stored data. L ) tracking time (T L ) or more, the number of unrecognized times is compared.

[0063] Next, the number of unrecognized times is compared. The judgment unit 33 judges whether the approximate number of unrecognized times of the temporarily stored data is the lowest data (R L ) unrecognized count (C L ) or more. L ) unrecognized count (C L), the determination unit 33 deletes the temporarily stored data. L ) unrecognized count (C L ) or more, compare the maximum number of targets.

[0064] Next, the maximum value of the number of objects is compared. The determination unit 33 determines whether the maximum value of the number of objects in the temporarily stored data is the lowest data (R L ) maximum number of targets (N L ) or more. The maximum number of temporary accumulated data objects is the lowest data (R L ) maximum number of targets (N L ), the determination unit 33 deletes the temporarily stored data. L ) maximum number of targets (N L ) or more, the lowest data (R L ) is deleted, and the temporarily stored data is stored in the additional learning data unit 34.

[0065] If the temporary accumulation data and the lowest data (R L ) and the maximum tracking time, number of unrecognized times, and number of targets are all the same, the data with the oldest video start date and time has higher priority. However, the data with the most recent video start date and time may also have higher priority.

[0066] As described above, the temporarily stored data and the lowest-ranked data are compared in a priority calculation process, and the data with the lower priority is deleted. However, the priority calculation is not limited to the above procedure. For example, the number of unrecognized events may be used as the index with the highest priority for comparison first. Also, if the lowest-ranked data does not satisfy the criteria for a certain index, the lowest-ranked data may be deleted at that point.

[0067] When temporarily stored data is stored and the data list is updated, the priority is determined anew for all data in the data list and the temporarily stored data according to the same priority calculation process, and the data list is updated. However, it is also possible to update only the lowest-ranked data.

[0068] (Storage process flowchart) FIG. 7 is a flowchart of the accumulation process.

[0069] In step S20, the temporary storage unit 32 performs temporary storage processing.

[0070] In step S21, the temporary storage unit 32 calculates an index from the image processing information associated with the temporarily stored data.

[0071] In steps S22 to S25, the decision unit 33 performs an accumulation decision process to decide whether or not to accumulate the temporarily accumulated data. In step S12, the judgment unit 33 judges whether the amount of data accumulated in the additional learning data unit 34 has reached a threshold. If the threshold has not been reached (if it is less than the threshold) (YES in step S22), the judgment unit 33 proceeds to step S26, accumulates temporarily accumulated data, and updates the data list. If the threshold has been reached (if it is equal to or greater than the threshold) (NO in step S22), the judgment unit 33 proceeds to step S23.

[0072] In steps S23 to S25, the decision unit 33 decides which of the temporarily stored data and the lowest-order data stored in the additional learning data unit 34 to store.

[0073] In step S23, the judgment unit 33 judges whether the tracking time of the temporarily accumulated data is equal to or longer than the tracking time of the lowest data. If the tracking time of the temporarily accumulated data is equal to or longer than the tracking time of the lowest data (YES in step S23), the judgment unit 33 proceeds to step S24. If the tracking time, which is the length of the temporarily accumulated data, is shorter than the tracking time of the lowest data (NO in step S23), the judgment unit 33 proceeds to step S28 and deletes the temporarily accumulated data.

[0074] In step S24, the judgment unit 33 judges whether the number of unrecognized times of the temporarily stored data is equal to or greater than the number of unrecognized times of the lowest data. If the number of unrecognized times of the temporarily stored data is equal to or greater than the number of unrecognized times of the lowest data (YES in step S24), the judgment unit 33 proceeds to step S28 and deletes the temporarily stored data.

[0075] In step S25, the judgment unit 33 judges whether the maximum number of objects in the temporarily accumulated data is equal to or greater than the maximum number of objects in the lowest-level data. If the maximum number of objects in the temporarily accumulated data is equal to or greater than the maximum number of objects in the lowest-level data (YES in step S24), the judgment unit 33 proceeds to step S26, deletes the lowest-level data from the additional learning data unit 34, accumulates the temporarily accumulated data in the additional learning data unit 34, and updates the data list. If the maximum number of objects in the temporarily accumulated data is less than the maximum number of objects in the lowest-level data (NO in step S24), the judgment unit 33 proceeds to step S28, and deletes the temporarily accumulated data.

[0076] In step S27, if the video data of the processed video signal Svi ends (YES in step S27), the accumulation determination process ends. If the processed video signal Svi contains video data (NO in step S27), the temporary accumulation process of step S20 is performed.

[0077] <Actions and Effects> By prioritizing data and using it to determine storage, low-priority data can be easily eliminated, and data for additional learning can be collected efficiently without requiring labor from workers.

[0078] It is also possible to select appropriate indices and prioritization according to the target objects to be tracked. For example, in this embodiment, the tracking time, the number of unrecognized instances, and the maximum number of targets are used as indices. Since a long tracking time results in a long video recording time, more data can be collected in the event of a malfunction, and this data is suitable for additional learning. Therefore, the tracking time is set as the first-priority judgment index. Next, since a large number of unrecognized instances increases the likelihood that a large amount of data will be included when a target object is overlooked, the number of unrecognized instances is set as the second-priority judgment index. Next, since a large maximum number of targets has the advantage of being able to collect more data because there are more people, the maximum number of targets is set as the third-priority judgment index.

[0079] In addition, because data for additional learning is stored in order of priority, data storage capacity is not wasted and communication costs for data communication can be reduced. Furthermore, by prioritizing data, learning data can be selected and used more efficiently, with higher priority data being used first.

[0080] [Variation 1] Modification 1 differs from the embodiment in that there is only one judgment step in the priority calculation process. Figure 8 is a diagram showing a flowchart of the accumulation process of Modification 1. In the following description, components that are the same as or equivalent to those in the above-described embodiment are given the same reference numerals, and their description will be simplified or omitted.

[0081] In the first modification, only one index is used for comparing the priority of the temporarily stored data with the lowest-ranked data, so the process of comparing the priority of the temporarily stored data with the lowest-ranked data is performed only once, at step 130.

[0082] <Actions and Effects> For example, if the data you want to collect is specific, such as a specific color or a specific range of heights, you can narrow down the indicators used to compare priorities, as in Variation 1, to more efficiently collect data for additional learning that suits your purpose.

[0083] [Variation 2] Modification 2 differs from the embodiment in that a single predetermined target object is identified in image processing. Fig. 9 is a diagram showing a flowchart of image processing of Modification 2. Here, the images of Figs. 2 and 3 are assumed. In the following description, components that are the same as or equivalent to those of the above-described embodiment are given the same reference numerals, and their description will be simplified or omitted.

[0084] In step S201, the image processing server 20 receives the video signal Sv from the camera unit 10. The video signal Sv is received via the input unit .

[0085] In step S202, the image processing server 20 performs recognition processing on a predetermined target object (for example, person A or person B) using AI (artificial intelligence) that uses a trained model for the video signal Sv.

[0086] In step S203, the recognition processing unit 22 determines whether or not a predetermined target object is recognized in the video. Here, object recognition can be performed, for example, on a frame-by-frame basis for the video signal. The recognition processing unit 22 also associates recognition information indicating whether the predetermined target object is "recognized" or "not recognized (unrecognized)" with the frame. If the predetermined target object is recognized (YES in step S203), the recognition processing unit 22 starts the tracking process in step S4. If the predetermined target object is not recognized (NO in step S203), the recognition processing unit 22 continues the recognition process (step S202).

[0087] The recognition processing unit 22 associates numerical information indicating the number of objects to be tracked that are recognized in a frame of the video signal with the frame. The association of the numerical information is continued while the image processing shown in FIG. 9 is being performed.

[0088] In step S204, the recognition processing unit 22 performs a tracking process of the recognized predetermined target object. Recognition of the object continues while the tracking process is being performed. Therefore, the recognition processing unit 22 associates recognition information indicating whether or not the predetermined object has been recognized with the frame. In addition, the recognition processing unit 22 associates tracking start information indicating the time point when tracking started with the frame.

[0089] In step S205, the recognition processing unit 22 determines whether the object being tracked has become unrecognized. If the recognition processing unit 22 does not recognize the predetermined target object, it associates recognition information indicating unrecognized with the frame. If the object being tracked is recognized (NO in step S205), the recognition processing unit 22 continues the tracking process. If the object being tracked is not recognized (YES in step S205), the process proceeds to step S206.

[0090] In step S206, the recognition processing unit 22 sets a determination condition for ending the tracking process based on the position of a predetermined target object in the frame immediately before the frame in which the target object has become unrecognized.

[0091] In step S207, the recognition processing unit 22 determines whether the determination condition is met. If the determination condition is not met (NO in step S207), the predetermined target object is in a state to be recognized, and therefore, a tracking process is performed. If the determination condition is met (YES in step S207), the recognition processing unit 22 ends tracking of the predetermined target object, and associates tracking end information indicating that tracking has ended with the frame.

[0092] In step S208, the recognition processing unit 22 determines whether image processing has been completed up to the end of the image included in the video signal Sv. If there is remaining image (NO in step S208), the recognition and tracking processing of the predetermined target object continues. If image processing has been completed up to the end of the image (YES in step S208), the image processing ends. Thereafter, the accumulation processing described in the embodiment is performed.

[0093] <Actions and Effects> In Modification 2, when the target object to be tracked is known in advance, the target object is recognized and tracked in a simple manner. This makes it possible to reduce the amount of data to be accumulated and efficiently collect data for additional learning.

[0094] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present invention. In this embodiment, a data collection system for additional learning including an image processing server and a storage server has been shown, but the present invention is not limited to this. The image processing server and the storage server do not necessarily need to be physically separate entities, and any system may be used that combines a configuration that realizes the functions of the image processing server (referred to as an "image processing function unit") with a configuration that realizes the functions of the storage server (referred to as an "storage processing function unit"). [Explanation of symbols]

[0095] 1. Data Collection System 10 Camera section 20 Image Processing Server 21, 31 Input section 22 Recognition processing section 23 Output section 24 Learning Department 25 Learning Data Section 30 Storage Server 32 Temporary storage unit 33 Judgment Department 34 Additional learning data section 35 Control Unit 40~42, 50~52 frames 43 Object 44a~44e, 45a~45d, 46a~46c, 54a~54e, 55a~55d, 56a~56c People A, B Specific person vf1, vf2, vfn, vfm, vfm+1 frame groups pf1, pf2, pfm, pfm+1 tracking end information ps1, ps2, psn, psm+1 tracking start information radius r1, r2

Claims

1. A data collection system for additional learning, comprising: an image processing function unit that performs image processing to recognize and track a target object included in a video captured of a predetermined area; and a storage processing function unit that stores data for additional learning of the image processing, The image processing function unit When it is recognized that a target object is included in the video, image processing is performed to add image processing information, which is information indicating that the target object is included, to the video data, which is data of the video; The storage processing function unit Temporarily storing the video data to which image processing information has been added, and calculating a predetermined index from the image processing information of the temporarily stored data, which is the temporarily stored data; performing a priority calculation process based on the index and a storage determination process for the temporarily stored data; The accumulation determination process includes: If the amount of data in the additional learning data section is less than a threshold, the temporarily stored data is stored in the additional learning data section; If the amount of data in the additional learning data section is equal to or greater than a threshold value, the temporarily stored data is compared with the data with the lowest priority among the data stored in the additional learning data section (hereinafter referred to as "lowest-order data") by the priority calculation process, and the data with the lower priority is deleted; The index includes any one of a tracking time, a number of unrecognized times, and a maximum number of targets; In the accumulation determination process, When the tracking time of the temporarily stored data is equal to or longer than the tracking time of the lowest data, and the number of times that the temporarily stored data has not been recognized is smaller than the number of times that the lowest data has not been recognized, deleting the temporarily stored data; A data collection system for incremental learning, comprising:

2. A data collection system for additional learning, comprising: an image processing function unit that performs image processing to recognize and track a target object included in a video captured of a predetermined area; and a storage processing function unit that stores data for additional learning of the image processing, The image processing function unit When it is recognized that a target object is included in the video, image processing is performed to add image processing information, which is information indicating that the target object is included, to the video data, which is data of the video; The storage processing function unit Temporarily storing the video data to which image processing information has been added, and calculating a predetermined index from the image processing information of the temporarily stored data, which is the temporarily stored data; performing a priority calculation process based on the index and a storage determination process for the temporarily stored data; The accumulation determination process includes: If the amount of data in the additional learning data section is less than a threshold, the temporarily stored data is stored in the additional learning data section; If the amount of data in the additional learning data section is equal to or greater than a threshold value, the temporarily stored data is compared with the data with the lowest priority among the data stored in the additional learning data section (hereinafter referred to as "lowest-order data") by the priority calculation process, and the data with the lower priority is deleted; The index includes any one of a tracking time, a number of unrecognized times, and a maximum number of targets; In the accumulation determination process, When the tracking time of the temporarily stored data is equal to or longer than the tracking time of the lowest data, If the number of unrecognized times of the temporarily stored data is equal to or greater than the number of unrecognized times of the lowest data, and if the maximum number of objects of the temporarily stored data is smaller than the maximum number of objects of the lowest data, delete the temporarily stored data. A data collection system for incremental learning, comprising:

3. In the accumulation determination process, 3. The data collection system for additional learning according to claim 1, wherein when the tracking time of said temporarily stored data is shorter than the tracking time of said lowest-order data, said temporarily stored data is deleted.

4. In the priority calculation process, 3. The data collection system for additional learning according to claim 1, wherein when the temporarily stored data and the lowest-level data have the same maximum tracking time, number of unrecognized times, and number of objects, the data with the oldest video start date and time is determined to have higher priority.

5. When the temporarily stored data is stored in the additional learning data section, 3. The data collection system for additional learning according to claim 1, wherein the data list in the additional learning data section is updated based on the priority calculation process.

6. A data collection method for additional learning, which performs image processing to recognize and track a target object included in a video captured in a predetermined area, and accumulates data for additional learning of the image processing, adding image processing information indicating that the target object is included in the video data, which is data of the video; a first step of temporarily storing data of the video data to which image processing information has been added, and calculating a predetermined index from the image processing information of the temporarily stored data; a second step of performing a priority calculation process based on the index and a storage determination process for the temporarily stored data, The accumulation determination process includes: If the amount of data in the additional learning data section is less than a threshold, the temporarily stored data is stored in the additional learning data section; If the amount of data in the additional learning data section is equal to or greater than a threshold value, the temporarily stored data is compared with the data with the lowest priority among the data stored in the additional learning data section (hereinafter referred to as "lowest-order data") by the priority calculation process, and the data with the lower priority is deleted; The index includes any one of a tracking time, a number of unrecognized times, and a maximum number of targets; In the accumulation determination process, When the tracking time of the temporarily stored data is equal to or longer than the tracking time of the lowest data, and the number of times that the temporarily stored data has not been recognized is smaller than the number of times that the lowest data has not been recognized, deleting the temporarily stored data; A data collection method for additional learning, comprising:

7. A data collection method for additional learning, which performs image processing to recognize and track a target object included in a video captured in a predetermined area, and accumulates data for additional learning of the image processing, adding image processing information indicating that the target object is included in the video data, which is data of the video; a first step of temporarily storing data of the video data to which image processing information has been added, and calculating a predetermined index from the image processing information of the temporarily stored data; a second step of performing a priority calculation process based on the index and a storage determination process for the temporarily stored data, The accumulation determination process includes: If the amount of data in the additional learning data section is less than a threshold, the temporarily stored data is stored in the additional learning data section; If the amount of data in the additional learning data section is equal to or greater than a threshold value, the temporarily stored data is compared with the data with the lowest priority among the data stored in the additional learning data section (hereinafter referred to as "lowest-order data") by the priority calculation process, and the data with the lower priority is deleted; The index includes any one of a tracking time, a number of unrecognized times, and a maximum number of targets; In the accumulation determination process, When the tracking time of the temporarily stored data is equal to or longer than the tracking time of the lowest data, If the number of unrecognized times of the temporarily stored data is equal to or greater than the number of unrecognized times of the lowest data, and if the maximum number of objects of the temporarily stored data is smaller than the maximum number of objects of the lowest data, delete the temporarily stored data. A data collection method for additional learning, comprising:

8. A data collection system for additional learning, comprising: an image processing function unit that performs image processing to recognize and track a target object included in a video captured of a predetermined area; and a storage processing function unit that stores data for additional learning of the image processing, The storage processing function unit Temporarily storing data and calculating predetermined indicators from the temporarily stored data; performing a priority calculation process based on the index and a storage determination process for the temporarily stored data; The accumulation determination process includes: If the amount of data in the additional learning data section is less than a threshold, the temporarily stored data is stored in the additional learning data section; If the amount of data in the additional learning data section is equal to or greater than a threshold value, the temporarily stored data is compared with the data with the lowest priority among the data stored in the additional learning data section (hereinafter referred to as "lowest-order data") by the priority calculation process, and the data with the lower priority is deleted; The index includes any one of a tracking time, a number of unrecognized times, and a maximum number of targets; In the accumulation determination process, The tracking time of the temporarily stored data is equal to or longer than the tracking time of the lowest data, and If the number of unrecognized times of the temporarily stored data is smaller than the number of unrecognized times of the least significant data, the temporarily stored data is deleted. A data collection system for incremental learning, comprising:

9. A data collection system for additional learning, comprising: an image processing function unit that performs image processing to recognize and track a target object included in a video captured of a predetermined area; and a storage processing function unit that stores data for additional learning of the image processing, The storage processing function unit Temporarily storing data and calculating predetermined indicators from the temporarily stored data; performing a priority calculation process based on the index and a storage determination process for the temporarily stored data; The accumulation determination process includes: If the amount of data in the additional learning data section is less than a threshold, the temporarily stored data is stored in the additional learning data section; If the amount of data in the additional learning data section is equal to or greater than a threshold value, the temporarily stored data is compared with the data with the lowest priority among the data stored in the additional learning data section (hereinafter referred to as "lowest-order data") by the priority calculation process, and the data with the lower priority is deleted; The index includes any one of a tracking time, a number of unrecognized times, and a maximum number of targets; In the accumulation determination process, When the tracking time of the temporarily stored data is equal to or longer than the tracking time of the lowest data, The number of times the temporarily stored data has not been recognized is equal to or greater than the number of times the lowest data has not been recognized, and If the maximum number of objects of the temporarily stored data is smaller than the maximum number of objects of the lowest-order data, the temporarily stored data is deleted. A data collection system for incremental learning, comprising:

Images