Degree-of-importance assessment system, degree-of-importance assessment device, and degree-of-importance assessment method

Abstract

A degree-of-importance assessment system (100) comprises: an identification means (101) for identifying a plurality of regions in one or more input images; a feature-amount calculation means (102) for calculating a feature amount for each region; and an assessment means (103) for generating, on the basis of the feature amount of each region, relationship information indicative of the relationship between the feature amounts of the regions, and assessing the degree of importance of each region on the basis of the relationship information.

Description

Importance determination system, importance determination device, and importance determination method

[0001] The present invention relates to an importance determination system, an importance determination device, and an importance determination method.

[0002] A technique for dividing and processing an input image when performing image analysis of the input image is known. For example, Patent Document 1 discloses an image analysis device including a partial image dividing unit that reprojects the input image in a plurality of different directions and divides it into a plurality of partial images, a feature extraction unit that extracts features from each of the partial images, an importance calculation unit that calculates an importance for each position of the input image based on the extracted feature amounts and a predetermined regression model, an attention point likelihood distribution calculation unit that calculates a likelihood distribution of attention points based on the calculated importance and a predetermined regression model, and an attention point calculation unit that calculates attention points based on the likelihood distribution of the attention points.

[0003] Japanese Patent Publication No. 2018-22360

[0004] Patent Document 1 describes calculating the importance for each position of an input image based on a predetermined regression model, but it would be useful if a technology could be provided that could determine the importance with higher accuracy.

[0005] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one of its objectives is to provide an importance determination system, an importance determination device, and an importance determination method that can determine importance with high accuracy.

[0006] An importance determination system according to one aspect of the present invention is an importance determination system that determines the importance of multiple regions within one or more input images, and includes: an identification means for identifying multiple regions within the one or more input images; a feature calculation means for calculating features of each region; and a determination means for generating relationship information indicating the relationship between the features of each region based on the features of each region, and determining the importance of each region based on the relationship information.

[0007] An importance determination device according to one aspect of the present invention is an importance determination device that determines the importance of multiple regions within one or more input images, and includes an identification unit that identifies multiple regions within the one or more input images, a feature calculation unit that calculates features of each region, a feature calculation means that calculates the features of each region, and a determination unit that generates relationship information indicating the relationship between the features of each region based on the features of each region, and determines the importance of each region based on the relationship information.

[0008] An importance determination method according to one aspect of the present invention is a method for determining the importance of multiple regions within one or more input images, comprising: a feature calculation means for identifying multiple regions within the one or more input images, calculating features of each region, and calculating the features of each region; a feature calculation means for generating relationship information indicating the relationship between the features of each region based on the features of each region; and determining the importance of each region based on the relationship information.

[0009] According to one aspect of the present invention, it is possible to determine the importance with high accuracy.

[0010] FIG. 1 is a block diagram showing an example of the configuration of an importance determination system according to a first embodiment. FIG. 2 is a flow diagram showing an example of the flow of an importance determination method according to the first embodiment. FIG. 3 is a block diagram showing an example of the configuration of an importance determination device according to the first embodiment. FIG. 4 is a block diagram showing an example of the configuration of an importance determination control system and a processing system according to a second embodiment. FIG. 5 is a schematic diagram showing an example of an area identified by an identification means in a second embodiment. FIG. 6 is a schematic diagram showing an example of a self-attention model. FIG. 7 is a flow diagram showing an example of a learning method for generating a trained model. FIG. 8 is a block diagram showing an example of the configuration of an importance determination system that executes a learning method. FIG. 9 is a schematic diagram showing an example of an area identified by an identification means in a third embodiment. FIG. 10 is a block diagram showing an example of the configuration of a computer.

[0011] [First Embodiment] A first embodiment of the present invention will be described in detail with reference to the drawings. This embodiment is the basis for the embodiments described below.

[0012] (Configuration of Importance Determination System) The configuration of the importance determination system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example configuration of an importance determination system 100 according to the first embodiment. The importance determination system 100 includes an identification unit 101, a feature calculation unit 102, and a determination unit 103, and determines the importance of multiple regions in one or more input images.

[0013] The input image may be captured by a camera connected to the importance determination system 100, or may be transmitted to the importance determination system 100 via a network. If the input image is captured by a camera, the number of cameras may be one or more. The camera may be a spherical camera, a panoramic camera, or the like.

[0014] The importance is an index used for predetermined processing based on the input image, and for example, the mode of processing may be changed based on the importance, or the flow of data being processed may be changed based on the importance. The predetermined processing based on the input image is not particularly limited, but may be, for example, processing to analyze an analysis target shown in the input image. The analysis target is not particularly limited, but may be, for example, a worker (person), a work device (object), and the behavior (movement) of the worker and the work device working at a construction site.

[0015] In this specification, "analysis" means detecting the occurrence of a target event in the analysis target. For example, if the analysis target is a worker (person), a work device (object), or the behavior (action) of the worker or the work device working at a construction site, the analysis result may be the detection result of the occurrence of an event such as inefficient work, procedural errors, or dangerous behavior.

[0016] In one embodiment, when importance is an index used for analyzing an input image, the importance may indicate the need for analysis. In this case, an event or object likely to occur that is the target of the analysis may be determined to be of high importance. Such "importance" may also be referred to as "attention level," "need for attention," or "danger level." Events of high importance include, but are not limited to, actions that are in accordance with a process, actions that deviate from a process, and actions that are highly dangerous. Objects of high importance include, but are not limited to, people and heavy machinery. The importance may also be determined based on whether or not they can be detected. For example, a person or object that is very small in the image and difficult to detect may be assigned a lower importance. The method of expressing importance is not particularly limited, and may be expressed as a binary value of "0" (low importance) or "1" (high importance), a multi-value of three or more values ​​(e.g., high, medium, low), or a continuous numerical value.

[0017] The specifying means 101 specifies a plurality of regions within one or more input images input to the importance determination system 101 .

[0018] The method for identifying multiple regions by the identification means 101 is not particularly limited, and it may identify regions corresponding to parts of the input image for which a predetermined importance is to be determined, it may identify regions surrounding objects detected by object detection processing on the input image, or it may identify regions obtained by dividing the input image at equal intervals.

[0019] The feature amount calculation means 102 calculates the feature amount of each region identified by the identification means 101. The method for calculating the feature amount is not particularly limited, and various known algorithms can be used.

[0020] The determination means 103 generates relationship information indicating the relationship between the feature amounts of each region based on the feature amounts of each region calculated by the feature amount calculation means 102, and determines the importance of each region based on the relationship information. In one aspect, the relationship information indicates the degree to which other regions are related to the importance of each region. In other words, the relationship information indicates the relationship between regions such that, for each region, the relationship is strong for regions necessary for determining the importance of that region, and the relationship is weak for regions not necessary for determining the importance of a specific region. Examples of such relationship information include attention weights used in attention mechanisms such as self-attention mechanisms.

[0021] As a result, the importance determination system 100 according to this embodiment can determine the importance with high accuracy. That is, since the relationship information is generated based on the feature amount of the input image, the importance can be determined using the relationship information according to the input image. As a result, the importance determination system 100 can determine the importance with high accuracy.

[0022] (Flow of Importance Determination Method) The flow of the importance determination method S100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing an example of the flow of the importance determination method S100 according to the first embodiment. In the example shown in Fig. 2, the importance determination system 100 executes the importance determination method S100.

[0023] In step S101, the identification unit 101 identifies multiple regions in one or more input images. In step S102, the feature calculation unit 102 calculates the feature amounts of each region. In step S103, the determination unit 103 generates relationship information indicating the relationship between the feature amounts of each region based on the feature amounts of each region, and determines the importance of each region based on the relationship information.

[0024] As described above, in the process control method S100 according to this embodiment, the importance of each region is determined using a trained model that generates relationship information based on the feature quantities of the input image. This allows the importance to be determined using relationship information corresponding to the input image, thereby enabling the importance to be determined with high accuracy.

[0025] (Configuration of Importance Determination Device) The configuration of the importance determination device 200 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the importance determination device 200 according to the first embodiment. The importance determination device 100 includes an identification unit 201, a feature calculation unit 202, and a determination unit 203, and determines the importance of multiple regions in one or more input images.

[0026] The identification unit 201 has the same function as the identification means 101, and identifies multiple regions in one or more input images. The feature calculation unit 202 has the same function as the feature calculation means 102, and calculates the feature amount of each region. The determination unit 203 has the same function as the determination means 103, and generates relationship information indicating the relationship between the feature amounts of each region based on the feature amounts of each region, and determines the importance of each region based on the relationship information.

[0027] The identification unit 201, the feature calculation unit 202, and the determination unit 203 may be computer devices in which processing is performed by a processor executing a program stored in a memory. For example, the identification unit 201, the feature calculation unit 202, and the determination unit 203 may be a single computer device, or may be a computer device group in which multiple computer devices operate in cooperation with each other, or a server device group in which multiple server devices operate in cooperation with each other. The importance determination device 200 can achieve the same effects as the importance determination system 100. Furthermore, some functions may be distributed to a cloud server.

[0028] Second Embodiment A second embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0029] 4 is a block diagram showing an example of the configuration of an importance determination system 100 and a processing system 1 according to the second embodiment. The importance determination system 100 according to this embodiment includes an identification unit 101, a feature amount calculation unit 102, a determination unit 103, and an analysis method control unit 104.

[0030] In this embodiment, the identification unit 101 identifies a plurality of regions whose positions are set in advance within the input image. Fig. 5 is a schematic diagram showing an example of regions identified by the identification unit 101 in this embodiment. The identification unit 101 divides one frame F of the input image into predefined regions and identifies a plurality of regions R. Here, the regions R do not need to be of a uniform size, and it is not necessary to identify regions from the entire frame F.

[0031] For example, the identifying unit 101 does not need to identify, as an area, a portion that is known in advance not to include the analysis target (for example, the sky or a building). For example, as shown in Figure 5, the identifying unit 101 may identify an area R from a portion other than the portion A corresponding to the sky.

[0032] Furthermore, for example, the identification unit 101 may change the size of the identified region depending on the characteristics of the input image or the analysis target. For example, as shown in Fig. 5, the identification unit 101 may make the size of the lower region R(2) showing the foreground larger than the size of the upper region R(1) showing the background, in accordance with the camera's angle of view. Furthermore, for example, the identified region may be larger for areas where large analysis targets may exist, and may be smaller for areas where small analysis targets may exist.

[0033] In this embodiment, the feature calculation means 102 calculates the feature of each region. The method for calculating the feature is not particularly limited, but in one aspect, the feature calculation means 102 may include, in the feature of each region, an estimation result of the type of object included in the region. The type of object indicates, for example, whether the object is a person, a machine, a vehicle, heavy machinery, or the like. In another aspect, the feature calculation means 102 may include, in the feature of each region, the position of the region within the input image. The representation format of the feature is not particularly limited, but may be, for example, a fixed-length vector.

[0034] For example, the feature of each region can be a fixed-length vector that combines position information indicating the position of the region within the input image and class information indicating the estimated type of object contained in the region.

[0035] The position information may be any information indicating the position of the region within the input image, and may be calculated according to the pixel position, for example, with the upper left corner of the input image as (0,0) and the lower right corner as (1,1). The position information may also include the size (width and height) of the region.

[0036] The class information may be any information indicating the estimated type of object contained in the region, such as the classification result (class classification) obtained by classifying each region using an object classification model. For example, an object classification model trained using training data such as ImageNet may be used as the object classification model. The representation format of the class information is not particularly limited, and may be, for example, a vector indicating the reliability of each identifiable type of object being contained in the region. For example, for region R(2) in FIG. 5, the information may be expressed as (car: 0.4, truck: 0.1, crane truck: 0.5, ..., person: 0).

[0037] Note that the features of each region calculated by the feature calculation unit 102 are not limited to those described above, and for example, features calculated using a learning model having a convolutional layer such as an Auto-Encoder may be used.

[0038] In this embodiment, the determination means 103 determines the importance of each region using a trained model M. In one aspect, the trained model M is a trained model that calculates a first matrix from input data combining features of each region and first parameters trained in advance by machine learning, calculates a second matrix from the input data and second parameters trained in advance by machine learning, calculates relationship information based on the first matrix and the second matrix, and calculates the importance of each region based on the relationship information. For example, the trained model M includes one or more layers that calculate the first matrix from input data combining features of each region and first parameters trained in advance by machine learning, one or more layers that calculate the second matrix from the input data and second parameters trained in advance by machine learning, one or more layers that calculate the relationship information based on the first matrix and the second matrix, and one or more layers that calculate the importance of each region based on the relationship information. The trained model M is not limited to this, but in one aspect, a self-attention model may be used.

[0039] Fig. 6 is a schematic diagram illustrating an example of a self-attention model. In the example illustrated in Fig. 6, the number of regions is 9, the number of dimensions of the feature is 1000, and the number of dimensions of the key (first matrix) and the query (second matrix) is 4. However, the number of regions and the number of dimensions are not limited to these.

[0040] 6, X is input data obtained by combining the features of each region, and is expressed as a (9, 1000) matrix in which the feature quantities of each of the nine regions, each with a dimension of 1000, are combined.

[0041] First, in the learning model M, the attention parameter W is set for the input data X. ｋ (first parameter) and attention parameter W ｑ (second parameter) respectively to obtain the key XW ｋ (first matrix) and query XW ｑ Ｔ (the second matrix) is obtained. The attention parameter W ｋ (first parameter) and attention parameter W ｑThe second parameter is a machine-learned parameter as described below, and is expressed as a (1000, 4) matrix. ｋ is a (9, 4) matrix, and the query XW ｑ Ｔ is a (4, 9) matrix.

[0042] Next, in the learning model M, an attention weight A is generated based on the following formula: ｉｎ indicates the number of dimensions of the feature. The attention weight A indicates which feature of each region is related to the importance of each region, and corresponds to relationship information.

[0043] Then, the importance determination result B(1,9) is calculated by calculating the sum of the attention weights A in the column direction. Each column of the importance determination result B(1,9) indicates the importance of each area.

[0044] As mentioned above, the attention parameter W ｋ (first parameter) and attention parameter W ｑ The second parameter is a machine-learned parameter. FIG. 7 is a flowchart illustrating an example of a learning method for generating a trained model M.

[0045] In one example, machine learning for generating the trained model M can be performed using an importance determination system 300 as shown in Fig. 8. The importance determination system 300 includes an identification unit 101, a feature calculation unit 102, a determination unit 103, a learning unit 105, and an analysis engine 106.

[0046] In step S1, training data is input to the training means 105. As training data, images with labels indicating analysis results may be used. The labels may further include a reward value used in reinforcement learning. Furthermore, multiple labels indicating analysis results may be attached to one training data.

[0047] For example, images such as those shown in FIG. 5 labeled "heavy machinery approaching (10)" and "transportation work (1)" can be used as training data for generating a trained model M to be applied to a construction site. Furthermore, for example, images labeled "packaging work (1)," "installation work (5)," and "screw tightening work (10)" can be used as training data for generating a trained model M to be applied to factory work. A high reward value may be set for an event with a high priority to be detected by analysis.

[0048] In addition, images containing multiple people can be used as learning data, with people detection as the analysis target, and the reward value can be set so that a high reward is given if an area containing people is selected (for example, +1 depending on the number of people included), and the reward is set to 0 if an area with nothing in it is selected.

[0049] In step S2, the learning means 105 calculates the parameters of the machine learning model M′ (for example, the attention parameter W ｋ (first parameter) and attention parameter W ｑ (second parameter)) is initialized.

[0050] In step S3, the learning means 105 determines whether or not there is learning data to be applied next, and ends learning if there is no learning data to be applied next.

[0051] In step S4, the identification means 101, feature calculation means 102, and judgment means 103 perform importance judgment in the same manner as the importance judgment system 100, except that they use learning data instead of the input image and use the machine learning model M' instead of the trained model M.

[0052] In step S5, the learning means 105 performs processing on the learning data, such as extracting only areas of high importance based on the importance of each obtained area, or performing processing to give high image quality to only areas of high importance and low image quality to other areas.

[0053] In step S6, the learning means 105 inputs the learning data processed in step S5 into the analysis engine 106 and identifies the analysis result. Note that multiple frames of images may be input to the analysis engine. The learning means 105 then calculates a reward value according to the obtained analysis result. That is, if the analysis result indicated by the label attached to the learning data is obtained, the learning means 105 may add the reward value set for the label.

[0054] In step S7, the learning means 106 calculates a parameter (attention parameter W ｋ (first parameter) and attention parameter W ｑ Reinforcement learning is performed by updating the second parameter (second parameter). The parameter updating method may be in accordance with a known reinforcement learning method.

[0055] As described above, by updating the parameters of the machine learning model M', a trained model M can be generated. Note that multiple types of trained models M may be generated. For example, different trained models M may be generated and used depending on the scene (construction site, factory work) or time of day (morning, noon, night), etc.

[0056] Next, a description will be given of the analysis method control means 104. The analysis method control means 104 controls the method of analyzing each region of the input image according to the importance of the region determined by the determination means 103.

[0057] Here, a processing system 1 that performs analysis processing of an input image will be described. As shown in Fig. 4, the processing system 1 includes one or more first processing units 20 and one or more second processing units 30. For ease of viewing, Fig. 4 illustrates a configuration with one first processing unit 20, but multiple first processing units 20 may be included.

[0058] Each first processing unit 20 is connected to, for example, a camera, a sensor such as LiDAR (Light Detection and Ranging), or the like, and acquires one or more input images from the camera, sensor, or the like. The input image is sufficient as long as the analysis target is included within the angle of view of the image. The analysis target is, for example, a worker (person), a work device (object), and the behavior (movement) of the worker and the work device working at a construction site.

[0059] The first processing unit 20 may be connected to a plurality of cameras, sensors, etc., and may acquire a plurality of input images. The first processing unit 20 may also acquire a plurality of input images from a single camera, etc.

[0060] The first processing unit 20 and the second processing unit 30 may each be configured by one or more computers. The first processing unit 20 and the second processing unit 30 can communicate with each other via a network NW and share the analysis processing of the input image. The network NW may be wireless or wired, and if wireless, may be a wireless communication system such as Wi-Fi, LTE, 4G, or 5G.

[0061] In one aspect, the first processing unit 20 may be an edge processing unit, and the second processing unit 30 may be a cloud processing unit. In this specification, "edge" refers to a location where data is collected. The first processing unit 20, which is an edge processing unit, is an information processing device (computer) or a group of information processing devices installed at or around a location where the analysis target is located (e.g., a construction site, a factory, etc.), and acquires input images from a camera, a sensor, etc. installed at the location where the analysis target is located. The first processing unit 20 may be integrated with a camera, a sensor, etc. Also, in this specification, "cloud" refers to a location where data processing, storage, etc. are performed. The second processing unit 30, which is a cloud processing unit, may be an information processing device (computer) or a group of information processing devices installed at a location that can provide large computational resources, such as a data center or a server farm. Note that the second processing unit 30 may be a processing unit located at a location connected to the first processing unit 20 via a network, and may be a computational resource connected to a base station such as 5G (5G network) (e.g., MEC (Multi-access Edge Computing)), a server installed in a field office, etc. (on-premises server), etc.

[0062] The first processing unit 20 may perform an analysis process on at least some of the one or more acquired input images to generate an analysis result. The first processing unit 20 may also calculate feature amounts for at least some of the one or more acquired input images and transmit the calculated feature amounts to the second processing unit 30 via the network NW. The first processing unit 20 may also transmit at least some of the one or more acquired input images to the second processing unit 30 via the network NW. When transmitting the feature amounts or the input images to the second processing unit 30, the first processing unit 20 may compress or encrypt the feature amounts or the input images before transmitting them to the second processing unit 30, or may transmit the feature amounts or the input images to the second processing unit 30 without compressing or encrypting them.

[0063] The second processing unit 30 receives the feature amount or input image transmitted from the first processing unit 20, and performs restoration processing as necessary and analysis processing.

[0064] The analysis process may include, for example, detection, identification, tracking, and time-series analysis of an analysis target (object, person) based on an input image. A learning model may be used for this analysis process. One or both of the first processing unit 20 and the second processing unit 30 may use the learning model.

[0065] The analysis method control means 104 controls the processing system 1 (that is, the first processing unit 20 and the second processing unit 30) as follows.

[0066] In one example, the analysis method control means 104 may control the first processing unit 20 to cut out areas of high importance from the input image, deliver them to the second processing unit 30, and discard the rest. In this way, for example, when the communication bandwidth between the first processing unit 20 and the second processing unit 30 is reduced, the bit rate can be reduced by delivering only the parts of high importance.

[0067] In one example, the analysis method control means 104 may control the first processing unit 20 to cut out areas of high importance from the input image and deliver them to the second processing unit 30, and analyze the remainder in the first processing unit 20. This allows the second processing unit 30 to analyze the areas of high importance from the input image using a high-precision model, and the first processing unit 20 to analyze the remaining areas using a low-precision model.

[0068] In one example, the analysis method control means 104 may control the first processing unit 20, the second processing unit 30, or both, to analyze only areas of high importance in the input image and discard the remaining areas. This makes it possible to focus the analysis on only the important areas when it is difficult to analyze all areas due to the computational load.

[0069] In one example, the analysis method control means 104 may control the first processing unit 20, the second processing unit 30, or both, to analyze only highly important areas of the input image using a high-precision model, and analyze the remaining areas using a low-precision model. In this way, when it is difficult to analyze all areas using a high-precision model due to the computational load, it is possible to analyze only the highly important areas using the high-precision model, and analyze the other areas using a low-precision model.

[0070] As described above, according to this embodiment, the area for determining importance is not limited to a fixed size, and importance can be determined for any size depending on the characteristics of the input image or the object of analysis.

[0071] In addition, since the importance of each region is determined using a trained model that generates relationship information based on the features of the input image, the importance can be determined using relationship information corresponding to the input image, allowing for highly accurate determination of importance.

[0072] Furthermore, by controlling the analysis method for each region according to the determined importance of each region, efficient analysis can be performed. In particular, when the camera is a spherical camera, a panoramic camera, or the like, and the amount of data is extremely large, by controlling so that only regions with high importance are transmitted from the first processing unit 20 to the second processing unit 30, it is possible to cope with a decrease in the network bandwidth.

[0073] In addition, in one example, the analysis method control means 104 may control the processing system 1 (i.e., the first processing unit 20 and the second processing unit 30) to divide the analysis of the data to be analyzed between the first processing unit 20 and the second processing unit 30. Note that the analysis method control means 104 may not cause the processing system 1 to analyze data to be analyzed that is determined not to require analysis.

[0074] The analysis of the data to be analyzed can be shared in various ways between the first processing unit 20 and the second processing unit 30. For example, the first processing unit 20 that has acquired the data to be analyzed performs all of the analysis processing of the data to be analyzed, the first processing unit 20 that has acquired the data to be analyzed performs some of the analysis processing and the second processing unit 30 performs the remaining analysis processing, or the first processing unit 20 performs the minimum necessary processing such as compression and the second processing unit 30 performs all of the analysis processing of the data to be analyzed. For example, the allocation method for analyzing the analysis target data may be selected from among a first allocation method in which the first processing unit 20 generates analysis results for the analysis target data, a second allocation method in which the first processing unit 20 calculates feature quantities of the analysis target data, transmits the feature quantities from the first processing unit 20 to the second processing unit 30, and generates analysis results from the feature quantities, and a third allocation method in which the first processing unit 20 transmits the analysis target data to the second processing unit 30, and generates analysis results from the analysis target data, depending on the computational capacity of the first processing unit 20. Criteria used to select the allocation method may include, in addition to computational capacity, computational cost, the importance of the analysis target data, the risk indicated by the analysis target data, the compression efficiency of each analysis target data, communication quality, and the like. By using these allocation methods appropriately, analysis processing can be performed efficiently depending on the situation.

[0075] Here, the analysis method control means 104 may select a sharing method according to the importance of each input image for each of the one or more input images acquired by each first processing unit 20. For example, an input image with a high importance may be analyzed quickly by being analyzed by the first processing unit 20 that acquired the input image, or an input image with a high importance may be analyzed by being analyzed by the second processing unit 30 so as to be analyzed with high accuracy.

[0076] In another aspect, the analysis method control means 104 may select a sharing method to switch between the first processing unit 20 and the second processing unit 30 analyzing the analysis target data based on a prediction of the processing load of the analysis target data in the first processing unit 20 and a prediction of the communication bandwidth between the first processing unit 20 and the second processing unit 30. The analysis method control means 104 may also determine a portion of the analysis target data to discard based on the predicted communication bandwidth. The analysis method control means 104 may also cause the first processing unit 20 and the second processing unit 30 to complement frames that were processed in the unit frame set before the switch from a state in which the analysis target data was not being processed to a state in which the analysis target data was being processed. The analysis method control means 104 may also buffer the analysis target data in one of the first processing unit 20 and the second processing unit 30 that is not analyzing the analysis target data, and when the processing unit that is not processing the analysis target data is switched to process the analysis target data, cause the processing unit to analyze the analysis target data using the buffered data. The process control means 102 may perform the discarding process, complementing process, and buffering process described above based on the importance of the data to be analyzed, the reliability of the processing of the data to be analyzed, the communication bandwidth allocated for transmitting the data to be analyzed, etc. The reliability is an index indicating the degree of confidence in the predicted analysis result, and may be, for example, a confidence value output from the trained model that performed the analysis.

[0077] Although the above description has been given of a configuration in which the importance determination system 100 is independent of each of the first processing units 20 and the second processing units 30, the present embodiment is not limited to this. For example, part or all of the importance determination system 100 may be provided in each of the first processing units 20, the second processing unit 30, or in each of the first processing units 20 and the second processing unit 30 in a distributed manner.

[0078] Although the second embodiment has been described above as process control system 100, the process control system 100 according to the second embodiment may be mounted on a single device to form a process control device. Furthermore, the operation of process control system 100 according to the second embodiment may be the process control method according to the second embodiment.

[0079] [Third Embodiment] A third embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first and second embodiments are given the same reference numerals, and their description will be omitted as appropriate.

[0080] In this embodiment, the identification unit 101 detects multiple objects included in one or more input images and identifies multiple regions by identifying regions corresponding to each of the detected objects. Fig. 9 is a schematic diagram showing an example of regions identified by the identification unit 101 in this embodiment. The identification unit 101 identifies multiple regions T1 and T2 by identifying regions corresponding to objects detected using an object detection model for one frame F of the input image. The regions corresponding to the objects are, for example, regions surrounding the objects.

[0081] In this embodiment, the feature amount calculation means 102 calculates the feature amount of each region in the same way as the feature amount calculation means 102 according to the second embodiment. At this time, the feature amount calculation means 102 may use, as the class information, a classification result obtained by classifying each region using an object classification model, as in the feature amount calculation means 102 according to the second embodiment. Alternatively, the feature amount calculation means 102 may use, as the class information, a classification result obtained when the identification means 101 detects an object in the input image by object detection.

[0082] As described above, according to this embodiment, the area for determining importance can be an area of ​​any size in which an object is detected, thereby enabling the importance to be determined efficiently.

[0083] Although the third embodiment has been described above as process control system 100, the process control system 100 according to the third embodiment may be mounted on a single device to form a process control device. Furthermore, the operation of process control system 100 according to the third embodiment may be the process control method according to the third embodiment.

[0084] [Fourth embodiment] A fourth embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first, second, and third embodiments are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0085] In this embodiment, the specifying unit 101 specifies a plurality of regions by specifying one or more regions in each of a plurality of input images input from different cameras.

[0086] In this embodiment, the feature calculation unit 102 calculates the feature of a region in each input image. Then, the determination unit 103 inputs input data that combines the feature of the regions in the multiple input images into the trained model M, thereby determining the importance of each of the regions in the multiple input images.

[0087] In this way, rather than inputting features for each input image into the trained model, input data that combines features from areas within multiple input images is input into the trained model, making it possible to perform importance judgments across multiple cameras.

[0088] Although the fourth embodiment has been described above as process control system 100, the process control system 100 according to the fourth embodiment may be mounted on a single device to form a process control device. Furthermore, the operation of process control system 100 according to the fourth embodiment may be the process control method according to the fourth embodiment.

[0089] The present disclosure is not limited to the above-described embodiments, and various modifications are possible. The technical scope of the present disclosure also includes embodiments obtained by appropriately combining the configurations, operations, and processes disclosed in different embodiments. Furthermore, the technical scope of the present disclosure also includes embodiments in which the order of the operations and processes disclosed in different embodiments is appropriately changed.

[0090] Each configuration according to the first to fourth embodiments may be realized by (1) one or more pieces of hardware, (2) one or more pieces of software, (3) a combination of hardware and software, or (4) any of the second. Each device, function, and process may be realized by at least one computer having at least one processor and at least one memory. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 10. For example, a program for implementing the processing control method according to the first to fourth embodiments may be stored in memory C2, and the processor C1 may read and execute program P stored in memory C2 to realize each function according to the first to fourth embodiments.

[0091] The program P includes a group of instructions that, when loaded into the computer C, causes the computer C to execute one or more of the functions described in the first to fourth embodiments. The program P is stored in the memory C2. The processor C1 can be, for example, a CPU (Central Processing Unit). The memory C2 can be, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), a flash memory, a Solid State Drive (SSD), or the like.

[0092] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0093] The present disclosure is not limited to the above-described embodiments. That is, the present invention can be applied to various aspects that can be understood by a person skilled in the art within the scope of the present disclosure. Note that part or all of the above-described embodiments can also be described as follows. However, the present invention is not limited to the aspects described below.

[0094] (Supplementary Note 1) An importance determination system for determining the importance of multiple regions within one or more input images, comprising: an identification means for identifying multiple regions within the one or more input images; a feature calculation means for calculating feature amounts of each region; and a determination means for generating relationship information indicating the relationship between the feature amounts of each region based on the feature amounts of each region, and determining the importance of each region based on the relationship information.

[0095] (Supplementary Note 2) The importance determination system described in Supplementary Note 1, wherein the determination means calculates a first matrix from input data combining features of each region and first parameters that have been machine-learned in advance, calculates a second matrix from the input data and second parameters that have been machine-learned in advance, calculates the relationship information based on the first matrix and the second matrix, and calculates the importance of each region based on the relationship information.

[0096] (Supplementary Note 3) The importance determination system according to Supplementary Note 1 or Supplementary Note 2, wherein the feature amount calculation means includes, in the feature amount of each region, an estimation result of the type of object included in the region.

[0097] (Supplementary Note 4) The importance determination system according to any one of Supplementary Notes 1 to 3, wherein the feature amount calculation means includes the position of each region in the input image as the feature amount of the region.

[0098] (Supplementary Note 5) The importance determination system according to any one of Supplementary Notes 1 to 4, wherein the identification means identifies the plurality of regions having at least two or more sizes whose positions within the one or more input images are preset.

[0099] (Supplementary Note 6) The importance determination system according to any one of Supplementary Notes 1 to 4, wherein the identification means detects a plurality of objects included in the one or more input images, and identifies the plurality of regions by identifying a region corresponding to each of the detected objects.

[0100] (Supplementary Note 7) The importance determination system according to any one of Supplementary Notes 1 to 6, wherein the identification means identifies the plurality of regions by identifying one or more regions within a plurality of input images input from different cameras.

[0101] (Supplementary Note 8) The importance determination system according to any one of Supplementary Notes 1 to 7, further comprising an analysis method control unit that controls an analysis method for each area in accordance with the importance of the area.

[0102] (Supplementary Note 9) An importance determination device that determines the importance of multiple regions in one or more input images, comprising: an identification unit that identifies multiple regions in the one or more input images; a feature calculation unit that calculates feature amounts of each region; a feature calculation means that calculates the feature amounts of each region; and a determination unit that generates relationship information indicating a relationship between the feature amounts of each region based on the feature amounts of each region, and determines the importance of each region based on the relationship information.

[0103] (Supplementary Note 10) The importance determination device described in Supplementary Note 9, wherein the determination unit calculates a first matrix from input data combining features of each region and first parameters that have been machine-learned in advance, calculates a second matrix from the input data and second parameters that have been machine-learned in advance, calculates the relationship information based on the first matrix and the second matrix, and calculates the importance of each region based on the relationship information.

[0104] (Supplementary Note 11) The importance determination device according to Supplementary Note 9 or 10, wherein the feature amount calculation unit includes, in the feature amount of each region, an estimation result of the type of object included in the region.

[0105] (Supplementary Note 12) The importance determination device according to any one of Supplementary Notes 9 to 11, wherein the feature amount calculation unit includes a position of each region within the input image in the feature amount of the region.

[0106] (Supplementary Note 13) The importance determination device according to any one of Supplementary Notes 9 to 12, wherein the identification unit identifies the plurality of regions having at least two or more sizes, the positions of which within the one or more input images being preset.

[0107] (Supplementary Note 14) The importance determination device according to any one of Supplementary Notes 9 to 12, wherein the identification unit detects a plurality of objects included in the one or more input images, and identifies the plurality of regions by identifying a region corresponding to each of the detected objects.

[0108] (Supplementary Note 15) The importance determination device according to any one of Supplementary Notes 9 to 14, wherein the identification unit identifies the plurality of regions by identifying one or more regions within a plurality of input images input from different cameras.

[0109] (Supplementary Note 16) The importance determination device according to any one of Supplementary Notes 9 to 15, further comprising an analysis method control unit that controls a method of analyzing each area in accordance with the importance of the area.

[0110] (Supplementary Note 17) An importance determination method for determining the importance of multiple regions in one or more input images, comprising: a feature calculation means for identifying multiple regions in the one or more input images; calculating a feature value for each region; generating relationship information indicating a relationship between the feature values ​​of each region based on the feature values ​​of each region; and determining the importance of each region based on the relationship information.

[0111] (Supplementary Note 18) An importance determination method as described in Supplementary Note 17, which calculates a first matrix from input data combining features of each region and first parameters that have been machine-learned in advance, calculates a second matrix from the input data and second parameters that have been machine-learned in advance, calculates the relationship information based on the first matrix and the second matrix, and calculates the importance of each region based on the relationship information.

[0112] (Supplementary Note 19) The importance determination method according to Supplementary Note 17 or 18, wherein the feature amount of each region includes an estimation result of the type of object included in the region.

[0113] (Supplementary Note 20) The importance determination method according to any one of Supplementary Notes 17 to 19, wherein the feature amount of each region includes the position of the region within the input image.

[0114] (Supplementary Note 21) The importance determination method according to any one of Supplementary Notes 17 to 20, wherein the plurality of regions having at least two or more sizes and having preset positions within the one or more input images are identified.

[0115] (Supplementary Note 22) The importance determination method according to any one of Supplementary Notes 17 to 20, wherein a plurality of objects included in the one or more input images are detected, and the plurality of regions are identified by respectively identifying regions corresponding to the detected objects.

[0116] (Supplementary Note 23) The importance determination method according to any one of Supplementary Notes 17 to 22, wherein the plurality of regions are identified by identifying one or more regions in each of a plurality of input images input from different cameras.

[0117] (Supplementary Note 24) The importance determination method according to any one of Supplementary Notes 17 to 23, wherein a method of analyzing each area is controlled according to the importance of the area.

[0118] (Supplementary Note 25) The above-described processing control system can also be expressed as follows.

[0119] An importance determination system that determines the importance of multiple regions in one or more input images, comprising at least one processor that executes: an identification process that identifies multiple regions in the one or more input images; a feature calculation process that calculates feature amounts of each region; and a determination process that generates relationship information that indicates a relationship between the feature amounts of each region based on the feature amounts of each region, and determines the importance of each region based on the relationship information.

[0120] The process control system may further include at least one memory that stores a program for causing the processor to execute the identification process, the feature calculation process, and the determination process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0121] (Supplementary Note 26) The above-described processing control system can also be expressed as follows.

[0122] An importance determination device that determines the importance of multiple regions in one or more input images, comprising at least one processor that executes an identification process that identifies multiple regions in the one or more input images, a feature calculation process that calculates feature amounts of each region, and a determination process that generates relationship information that indicates a relationship between the feature amounts of each region based on the feature amounts of each region, and determines the importance of each region based on the relationship information.

[0123] The process control system may further include at least one memory that stores a program for causing the processor to execute the identification process, the feature calculation process, and the determination process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0124] REFERENCE SIGNS LIST 1 Processing system 10 Camera 20 First processing unit 30 Second processing unit 100 Importance determination system 101 Identification means 102 Feature amount calculation means 103 Determination means 104 Analysis method control means 104 M Trained model

Claims

1. 1. An importance determination system for determining importance of a plurality of regions in one or more input images, comprising: means for identifying a plurality of regions within the one or more input images; A feature amount calculation means for calculating a feature amount of each region; An importance determination system comprising: a determination means for generating relationship information indicating a relationship between the feature amounts of each region based on the feature amounts of each region, and determining the importance of each region based on the relationship information.

2. 2. The importance determination system of claim 1, wherein the determination means calculates a first matrix from input data combining features of each region and a first parameter that has been machine-learned in advance, calculates a second matrix from the input data and a second parameter that has been machine-learned in advance, calculates the relationship information based on the first matrix and the second matrix, and calculates the importance of each region based on the relationship information.

3. 2. The importance determination system according to claim 1, wherein the feature amount calculation means includes in the feature amount of each region an estimation result of a type of object contained in the region.

4. 2. The importance determination system according to claim 1, wherein the feature amount calculation means includes in the feature amount of each region the position of the region within the input image.

5. 2 . The importance determination system according to claim 1 , wherein the specifying means specifies the plurality of regions having at least two or more sizes, the positions of which in the one or more input images are preset.

6. The importance determination system according to claim 1 , wherein the specifying means detects a plurality of objects included in the one or more input images, and specifies the plurality of regions by specifying a region corresponding to each of the detected objects.

7. The importance determination system according to any one of claims 1 to 6, wherein the identification means identifies the plurality of regions by identifying one or more regions within a plurality of input images input from different cameras.

8. An importance determination device for determining importance of a plurality of regions in one or more input images, comprising: an identification unit for identifying a plurality of regions within the one or more input images; A feature amount calculation unit that calculates a feature amount of each region; A feature amount calculation means for calculating a feature amount of each region; An importance determination device comprising: a determination unit that generates relationship information indicating a relationship between the features of each region based on the features of each region, and determines the importance of each region based on the relationship information and the features of each region.

9. The importance determination device of claim 8, wherein the determination unit calculates a first matrix from input data combining features of each region and a first parameter that has been machine-learned in advance, calculates a second matrix from the input data and a second parameter that has been machine-learned in advance, calculates the relationship information based on the first matrix and the second matrix, and calculates the importance of each region based on the relationship information.

10. 1. A method for determining importance of a plurality of regions in one or more input images, comprising: Identifying a plurality of regions within the one or more input images; Calculate the feature values ​​for each region, A feature amount calculation means for calculating a feature amount of each region; An importance determination method, comprising: generating relationship information indicating a relationship between the feature amounts of each region based on the feature amounts of each region; and determining the importance of each region based on the relationship information and the feature amounts of each region.