Activity status determination system, determination device, and service provision system

Abstract

According to the embodiment, an activity status determination system is provided that allows a stable activity status determination status to be visually confirmed easily with high accuracy. [Solution] In an embodiment, multiple original pixels (Pif) are sequentially analyzed per multiple number of frames (n) to become reference pixels (DPk). A two-dimensional array of reference pixels (DPkr) becomes a reference frame (DFd). Each reference pixel (DPd) includes a time-series average luminance value and a luminance standard deviation relative to this average luminance value. It also includes a threshold value for determining the luminance difference between the luminance of the original pixel (Pif) in the time series direction, based on the luminance standard deviation and the average value of other luminance standard deviations within the frame, for each frame. Areas of reference pixels (DPkr) where the luminance difference determination threshold value changes in the time series direction are determined as variable pixel areas, and the determination results for each of the multiple areas are converted into display pixels for multiple areas of a single display frame.

Description

[Technical Field]

[0001] The embodiments of the present invention include an activity status determination system, a determination device, and a service providing system. Mu Regarding. [Background technology]

[0002] For example, at work sites in factories and logistics centers, there is a demand for systems that use cameras to detect the activity status (active / inactive) of trucks, workers (people), forklifts, etc.

[0003] In systems that detect a person's activity state, cameras generally capture images from the side or from diagonally above so that the entire person can be captured. Activity status is then determined by motion detection processing using the captured image data.

[0004] However, if there are multiple locations that need to be monitored simultaneously, it is necessary to install cameras at each location.For example, there may be multiple monitoring locations, such as the location where packages are brought in, the location where packages are moved, the location where workers are waiting, the location where packages are sorted, and the location where packages are taken out.

[0005] To address this issue, omnidirectional cameras (360-degree cameras: cameras with 360-degree lenses) or cameras with fixed angles of view are sometimes installed from directly above so that the entire area can be captured. However, the angle of view captured from above is different from the angle of view captured from the side or from diagonally above. This means that a person's entire body is not captured (for example, the lower half of the body is not captured), making it unclear whether the person is standing or sitting. As a result, even if a person is detected, it is difficult to determine whether the person is performing a task (whether the task can be performed or not).

[0006] On the other hand, there is a method that uses an artificial intelligence system to learn the characteristics of people's images through deep learning and detect people directly.When using this system, it is necessary to acquire and learn a huge number of people's images, because the orientation and characteristics of the people captured in the images vary depending on the location of the place to be monitored.

[0007] It is also possible to consider a system that determines activity status simply by detecting motion in an image, but this has the problem of the motion detection function reacting to brightness fluctuations in the image caused by things other than human activity, resulting in false detections. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-250892 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-204704 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-113543 Summary of the Invention [Problem to be solved by the invention]

[0009] Due to the labor shortage in recent years, truck drivers are increasingly performing multiple tasks (working hours) such as loading and unloading. On the other hand, there is a trend toward enacting regulations on continuous working hours (including driving time) for long-distance drivers in order to prevent accidents.

[0010] In view of the above circumstances, an activity status determination system is required to determine the activity status of a driver stably with high accuracy.

[0011] To ensure a highly accurate and stable activity status determination system, - To reliably identify workers and monitor their movements (activities), - There are various types of work sites, such as warehouse locations, warehouse sizes, and the width and orientation of loading / unloading entrances and exits. The activity status assessment system can be flexibly introduced to these work sites. - Minimizing the cost of the activity status assessment system; The activity status determination system has high determination capability and is reliable. -In addition, the activity status determination status can be easily confirmed visually, etc. are required.

[0012] Therefore, according to the embodiment of the present invention, an activity status determination system, a determination device, and a service providing system are provided that can easily check the activity status determination status with high accuracy and stability. M The purpose is to provide. [Means for solving the problem]

[0013] One embodiment is an activity status determination system having a data processing unit including a luminance variation detection unit and an imaging unit, A captured frame is defined as an original frame (Fh) that contains multiple original pixels (Pif) with different areas within the frame. A reference pixel (DPkr) having information obtained by sequentially analyzing each of the plurality of original pixels (Pif) in the original image frame (Fh) per a predetermined number of frames (n) in a time series direction is generated, and a frame in which the reference pixels (DPkr) are two-dimensionally arranged is defined as a reference frame (DFd); When one of the reference pixels (DPkr) is focused on, The original pixel (Pif) includes a determination threshold value for a luminance difference in a time series direction, The brightness fluctuation detection means a means for obtaining determination result data indicating an area of ​​reference pixels (DPkr) corresponding to the change in level as an area of ​​changed pixels when the determination threshold value of the luminance difference changes in the time series direction; The imaging means An activity status determination system is provided, which includes a means for converting each determination result data detected at the reference pixels (DPkr) in different areas of the plurality of reference frames in the time series direction into display pixels in an area of ​​one display frame that corresponds to the different areas. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an example of the overall configuration of one embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing a specific example of the configuration of the peripheral information processing unit in FIG. [Figure 3] FIG. 3 is a flowchart showing the general operation of the activity state determination unit shown in FIGS. [Figure 4] FIG. 4 is a diagram showing an example of a 360-degree camera and an image captured by the camera. [Figure 5] FIG. 5 is a diagram showing an example of an image captured by a 360-degree camera after being dewarped (cut out). [Figure 6] FIG. 6 is an explanatory diagram showing how reference frames (DFd) are generated sequentially from a plurality of original frames (Fh) in the time series direction. [Figure 7] FIG. 7 is an explanatory diagram for explaining the contents of the reference pixels (DPkr) forming the reference frame (DFd) in the figure. [Figure 8] FIG. 8 is an explanatory diagram showing thresholds for determining stable pixels, fluctuating pixels, and over-fluctuating pixels, which are generated using data in a reference frame (DFd) at a certain point in time. [Figure 9] FIG. 9 is an explanatory diagram showing an example of conditions for determining the three types of reference pixels of the stable pixel system, the changing pixel system, and the over-changing pixel system shown in FIG. [Figure 10] FIG. 10 is an explanatory diagram showing an example of fluctuations in the luminance difference determination threshold (BTi) for stable pixels, fluctuating pixels, and over-fluctuating pixels. [Figure 11]Figure 11 is an explanatory diagram that explains the process by which the three types of pixels, stable pixels, variable pixels, and over-variable pixels, are identified for each pixel, and then the variable pixel area is changed into image data for display and image identification processing. [Figure 12] FIG. 12 is an explanatory diagram showing a series of steps from image data conversion of the variable pixel region in FIG. 11 to state determination by image recognition, using a specific image example. [Figure 13] FIG. 13 is an explanatory diagram showing an example of the second mask processing. [Figure 14] FIG. 14 is an explanatory diagram when image data for display in the time series direction is synthesized using the weighted average method. [Figure 15] FIG. 15 is an explanatory diagram showing an example of a software block configuration in another embodiment. [Figure 16] FIG. 16 is an explanatory diagram showing an example of a block configuration of hardware in yet another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments will be described with reference to the drawings. 1, reference numeral 10 denotes a 360-degree camera, and image data 11 captured by this 360-degree camera is input to a capture unit 100. The capture unit 100 converts the image data 11 into dewarped input image data 20.

[0016] The input image data 20 from the capture 100 is input to a pre-processing unit 301 of the activity state determination unit 300 and also to a peripheral information processing unit 500 .

[0017] The activity state determination unit 300 processes image data and detects an area in the image where, for example, a worker is working. In response to this, the surrounding information processing unit 500 acquires information that is highly correlated with the worker's activity. Examples of highly correlated information include information that detects that a truck or forklift has arrived near a work area, or information that detects that it has departed from near a work area. Further examples of highly correlated information include information that identifies the work area where work is being performed, and information that detects an increase or decrease in the amount of cargo in the work area.

[0018] The activity state determination unit 300 includes a pre-processing unit 301, a luminance variation detection unit 302, and an image classification unit 303. These are connected to a memory 311 and an arithmetic processing unit 313, and can operate in conjunction with each other under commands from a system controller 1000. The system controller 1000 controls each block of the entire system. Furthermore, the pre-processing unit 301, the luminance variation detection unit 302, and the image classification unit 303 each have a processing circuit and / or a dedicated program according to their respective roles. The cooperative operations between the blocks include, for example, temporary data storage, data processing such as reading and writing data, and arithmetic processing for performing calculations and processing using data. The activity state determination unit 300 may be referred to as an activity state determination device, activity state determiner, activity state determination circuit, etc. The preprocessing unit 301 may be referred to as a preprocessing unit, preprocessor, preprocessing circuit, etc. The brightness fluctuation detection unit 302 may be referred to as a brightness fluctuation detection device, brightness fluctuation detector, brightness fluctuation detection circuit, etc. The image identification unit 303 may be referred to as an image identification device, image classifier, image identification circuit, etc.

[0019] The pre-processing unit 301 acquires image data of an area suitable for grasping the activity state, that is, cuts out the image data.

[0020] The extracted image data is input sequentially to the brightness fluctuation detection unit 302 as original image frame data including pixel data. In this embodiment, pixel data and frame data are processed, but for the sake of simplicity, in this specification, pixel data is simply referred to as "pixels" and frame data is simply referred to as "frames." The brightness fluctuation detection unit 302 collects information from multiple original frames (Fh, described later) in a time series, processes the data, and generates a reference frame (DFd, described later). The original frame (Fh) has multiple original pixels (e.g., 320 pixels x 240 pixels) (Pif, described later) within one frame. Corresponding to these original pixels, the reference frame has multiple reference pixels (DPkf, described later) within one reference frame. These reference pixels contain various information generated from the original pixels.

[0021] As described above, in this embodiment, the plurality of reference pixels in the reference frame are not simply synthesized data. The reference frame contains various information related to the plurality of original image frames in the time series direction. The information contained in the reference pixels is as follows: *The average brightness value of multiple (n) original pixels in the time series direction, *The brightness standard deviation of the above multiple (n) original pixels, *Furthermore, there is an "environment-adaptive luminance standard deviation" that is calculated based on the luminance standard deviation of the entire original frame and can respond to environmental changes (such as changes in ambient brightness) of the entire original frame. *In addition, there is a "brightness difference judgment threshold" that uses this "environment-adaptive brightness standard deviation." *Then, the time-series average value and standard deviation of the "brightness difference determination threshold" are used as "data used to determine pixel type." Pixel types are stable pixels, fluctuating pixels, and over-fluctuating pixels. Detecting the fluctuation state of the luminance difference of the luminance of the original pixels corresponds to detecting the state in which, for example, a worker or a work vehicle (which may be called a working moving body) is active.

[0022] Although the above description has been given with respect to an original pixel as a single pixel unit, a pixel block containing multiple original pixels may also be treated as a single unit. An original pixel unit is, for example, a unit in the case where one original image frame is (320 pixels x 240 pixels), and a pixel block unit is, for example, a unit in the case where one original image frame is (40 blocks x 30 blocks) and one block is (8 pixels x 8 pixels). When processing this type of pixel block, the average luminance value of (8 pixels x 8 pixels) = 64 pixels may be treated as the representative luminance of one pixel block, or, for example, the luminance of the pixel with the highest luminance may be treated as the representative luminance of one pixel block. Alternatively, a majority vote of the 64 luminance values ​​may be taken and the most common luminance value may be treated as the representative luminance.

[0023] In this specification, when we refer to "luminance fluctuations of original pixels in the time series direction (or fluctuations in the luminance difference of original pixels)", we mean luminance fluctuations both when using pixel units and when using pixel block units.

[0024] The outline of the functions of the brightness fluctuation detection unit 302 will be summarized and explained below in 1), 2), 3) and 4).

[0025] 1) The brightness fluctuation detection unit 302 generates a reference frame (DFd) containing various data (such as an environment-adaptive brightness standard deviation) generated using information from a time-series original image frame (Fh). Let the frame frequency of the original image frame be (F). The reference frame (DFd) has a frame frequency (F) and is obtained one after another for each of a number (n) of original image frames (n is an integer) in the time-series direction. The reference frame (DFd) has the same frequency (F) as the frame frequency (F) of the original image frame.

[0026] 2) The brightness fluctuation detection unit 302 performs a first mask process that roughly classifies the image capture area. The first mask process is a mask process with the following purpose. That is, within a cropped image, it is possible to determine in advance the areas where there is a "luminance change among multiple pixels in the time series direction," i.e., the active area or working area, and the other areas (non-active area or non-working area) by designing and setting them. Therefore, a first masking process using a mask signal is executed on the previously set non-working area. The image areas not required for this first masking process may be set using preset values, or may be set manually by a supervisor at the work site while observing the site. This setting leaves the active area or working area, improving the efficiency of the luminance variation detection process.

[0027] 3) The brightness fluctuation detection unit 302 acquires data of the brightness fluctuation detection process result indicating the working status of the worker from the reference frame (DFd). The brightness fluctuation detection process method will be described in more detail later.

[0028] In the brightness fluctuation detection process, the brightness standard deviation is used to set a brightness difference judgment threshold (the environment-adaptive brightness standard deviation) to detect whether or not the brightness (change) of an original pixel is fluctuating among n consecutive original pixels in the time series direction (the definition of the brightness difference judgment threshold is explained in more detail later).

[0029] It is determined whether or not the threshold value for determining the luminance difference between the n consecutive original pixels in the time series direction (the aforementioned environment adaptive luminance standard deviation) fluctuates.

[0030] In the detection process, if a pixel is stable, there is almost no fluctuation in brightness, and therefore the threshold value for determining the brightness difference between the previous and next original pixels remains low and does not fluctuate. In the detection process, if a pixel is found to be a variable pixel, the threshold value for determining the luminance difference between the previous and next original pixels repeatedly changes from high to low drastically in the time series direction. In the detection process, if the pixel is found to be an over-fluctuation pixel, the determination threshold value for the luminance difference between the previous and next original pixels remains high and does not fluctuate. 4) The brightness fluctuation detection unit 302 performs imaging processing by replacing (or converting) the area of ​​fluctuating pixels where the brightness difference judgment threshold changes over time with display pixels (which may also be called display brightness values). At this time, a second masking process is also performed. The second masking process is a masking process for narrowing down to only specific work locations within the work site area. In other words, a preset area is masked to prevent unnecessary judgment result data from being converted into display pixels.

[0031] The display pixels corresponding to the above-mentioned varying pixels are obtained from multiple original image frames in a time series direction. Moreover, because the moving object is moving, the display pixels are observed in different pixel areas within the frame space of the display frame when viewed in a time series direction. In this system, display pixels generated (detected) more recently are weighted heavily, and display pixels generated (detected) earlier are weighted lightly, and the display pixels are combined to generate the final display frame. This final display frame (which may be called a varying pixel image) is a frame image of the brightness fluctuation state in the video. Therefore, by using this for image recognition, it is possible to extract the various video states that the user wants to recognize, classify situations, or extract information.

[0032] The final display frame generated as described above is input to the image classification unit 303. The image classification unit 303 may perform more accurate image classification using, for example, a neural network on the data resulting from the brightness variation detection, and may also output a classification class for the image. The classification class is the state (or the content of the image scene, etc.) classified by the neural network, such as whether there is a person present or not, whether there is a forklift present or not, whether work is being done or not, etc.

[0033] For example, the width of the moving trajectory of the moving object in the image of the moving pixels appearing in the final display frame may vary depending on the size of the moving object. Furthermore, depending on the relationship between the moving object's moving speed and the number of n frames (frame frequency), the moving trajectory image may appear as a line or a dotted line. These images, along with the imaging conditions, may be trained in advance by a neural network to obtain a discrimination class. Furthermore, this discrimination class may be used as service information.

[0034] The determination result from the activity state determination unit 300 is input to the post-processing unit 700. Peripheral information is input to the post-processing unit 700 from the peripheral information processing unit 500. This peripheral information processing unit 500 is also provided with a memory 511 and an arithmetic processing unit 513, which can operate in conjunction with each other in response to commands from the system controller 1000.

[0035] The post-processing unit 700 can provide high reliability to the determination result of the activity state by using information correlated with the determination of the activity state. For example, in a scene where luggage is delivered by truck, if the peripheral information processing unit 500 detects that the luggage in the luggage storage area is gradually increasing or that the space in the luggage storage area is gradually decreasing while the worker is working, it can be determined that the worker's activity state is normal. It is also possible to determine the work efficiency based on the speed of the increase or decrease in the amount of luggage (increase or decrease in the space in the luggage storage area).

[0036] Conversely, there may be a scenario where the number of items in the storage area does not increase despite the presence of workers. In this case, it is possible that the items are not being unloaded, or that the items have been transported to a different location than the original unloading location. Furthermore, a system failure may also be suspected.

[0037] The output data of the post-processing unit 700 is transmitted as service data via a network and can be monitored on various user interfaces 900 (such as personal computers, servers, and smartphones). This system allows the output data to be used as secondary service data.

[0038] 2 shows an example of the internal configuration of the peripheral information processing unit 500. This example shows a case where this system is used, for example, in a management system for truck berths in a logistics warehouse. However, this system can also be applied to material storage areas at construction sites, temporary storage areas for tools and work machines, temporary storage areas for tools and machines at civil engineering sites, parking lots, areas for repair and inspection of cars and airplanes, product delivery areas and waste delivery areas at supermarkets, etc.

[0039] The pre-processing unit 401 can obtain, for example, cropped image data of the locations of trucks, forklifts, tools, etc. at the departure and arrival point, or of waiting rooms when workers are not working. Image data from a separate dedicated camera may also be used here. The image data from the pre-processing unit 401 is input to the object detection unit 402. The object detection unit 402 has pre-trained learning data and detects the trucks, forklifts, workers, etc. If a truck, forklift, or worker is detected, this has a strong correlation with the activity status determination process performed by the activity status determination unit 300, ensuring the reliability of the activity status determination results. While the use of image data has been described here, the output of other sensors (e.g., sound detection, human body detection, temperature detection, light detection, vibration detection, etc. (which may be referred to as IoT)) may also be used.

[0040] The preprocessing unit 601 obtains image data of, for example, a warehouse site near a truck arrival and departure point. This image data is input to the area extraction unit 602. Here, for example, when unloading or loading work is being carried out, it is possible to monitor the space (area) where luggage is stored. Furthermore, by monitoring the area within the warehouse, it is possible to manage whether unloading and loading are being carried out reliably. Furthermore, the image of the luggage storage space may be used as information for setting the mask area.

[0041] If an increase or decrease in cargo volume is accurately detected in images of unloading or loading operations, service data can be sent to other trucks waiting within the warehouse premises to be unloaded or loaded at the appropriate time.

[0042] 3 shows an example of a data processing procedure in the activity state determination unit 300. The data processing unit (including the capture unit 100, preprocessing unit 301, brightness variation detection unit 302, and image identification unit 303), memory unit 311, and calculation processing unit 313 cooperate with each other via the system controller 1000 to perform data processing.

[0043] That is, in step AS1, a process for acquiring image data from the camera 10 is executed. In step AS2, a dewarping process (a process for cutting out the area to be analyzed) is executed (steps AS1 and AS2 are processes performed by the capture unit 100 shown in FIGS. 1 and 2). In this step, a first masking process is executed. The first masking process is performed, for example, on image areas that do not require detection processing in advance. This unnecessary image area may be set using a preset value, or may be set manually by a manager at the work site while observing the site. In step AS3, the process of detecting brightness fluctuation pixels and brightness fluctuation pixel blocks is executed sequentially in the order of multiple original image frames in time series. The software and / or hardware that executes step AS3 and steps AS4, AS5, and AS6, which will be described subsequently, are included in the brightness fluctuation detection unit 302 in FIG.

[0044] The result of the detection in step AS3 (when the pixel type determination threshold changes over time) is used in the next step AS4 to image the results of the brightness variation detection process. In the image processing, the area of ​​the varying pixels is replaced with pixels for display, and the final display frame is generated.

[0045] In the next step AS5, a second masking process is performed on the final display frame, which is the result of the brightness fluctuation detection process. That is, in step AS5, only specific work locations within the work site area are narrowed down to be the brightness fluctuation detection target area, and the second masking process is performed on the other areas.

[0046] Furthermore, in step AS6, a synthesis process is performed on the brightness fluctuation detection processing results (the previous final display frame). That is, the imaging data of the brightness fluctuation detection results that have been previously masked is synthesized with the imaging data of the brightness fluctuation detection results that have been currently masked. Here, synthesis is performed using a simple averaging method or a weighted averaging method. (Steps AS3 to AS6 are processes performed by the brightness fluctuation detection unit 302 described in Figures 1 and 2 and in 1)-4) above.)

[0047] In the next step AS7, processing is performed by the image classification unit 303. Then, in step AS8, a final determination of the active state is made (this final determination processing is performed by the post-processing unit 700 in FIGS. 1 and 2).

[0048] Figure 4 shows the 360-degree camera 10 shown in Figures 1 and 2, and an example of a 360-degree image taken by this camera 10 looking down in all directions. A truck depot is imaged in front (towards the top of the image on the paper), and the truck bed is imaged from behind on the right. Using the 360-degree camera 10 provides a wide imaging range, makes it possible to extract images from multiple locations, and is less expensive than systems using multiple cameras.

[0049] FIG. 5 shows the dewarped image (right side of the figure) obtained by cutting out the upper half of the above 360-degree image (left side of the figure). The dewarped image has its curvature corrected (pixel interpolation processing) to become an image similar to an image captured by a camera with a normal angle of view (standard image). This processing may involve resizing. Here, the first mask processing is performed.

[0050] First, the basic technology of the embodiment will be described with reference to Figures 6 to 10. The inventors of the present application have developed a technology for tracking and detecting the location of a moving object based on brightness fluctuations (appearance and disappearance) that occur when the moving object moves within a monitored area. In this case, the "environment-adaptive brightness standard deviation" described above is used to facilitate imaging of the moving object's trajectory. This "environment-adaptive brightness standard deviation" is capable of robustly tracking fluctuations in brightness of the entire image and noise, and is used to set a "brightness difference determination threshold" for determining the presence or absence of brightness in each pixel. The finally determined "brightness difference determination threshold" is used to obtain data for determining pixel type (hereinafter referred to as "data used for pixel type determination").

[0051] Here, the "data used for determining pixel type" is obtained in the following steps: As mentioned above, the average luminance value and standard deviation of luminance of a plurality of (n) original pixels in the time series direction are used. Furthermore, an "environment-adaptive luminance standard deviation" is calculated based on the luminance standard deviation of the entire original image frame, and corresponds to at least environmental changes in brightness of the entire original image frame. The "environment-adaptive luminance standard deviation" is used to determine the "luminance difference determination threshold." The average value and standard deviation of the "brightness difference determination threshold" are then used as "data used for pixel type determination." The environmental change may be due to the color of the environment (for example, the color of a sunset in the evening).

[0052] The characteristics of the reference pixels will be further described with reference to Figures 6 and 7. In this embodiment, the area in which a moving object moves is captured by a camera, and original image frames Fh (h = 1, 2, 3, ... n) (shown on the left side of Figure 6) are acquired sequentially in time series. An original image frame (Fh) is defined as an original image frame (Fh) that includes multiple original pixels (Pif) in different areas (or arranged two-dimensionally) within that frame.

[0053] Next, n frames of original image frames (Fh) (h=1, 2, 3,...n) acquired in chronological order are sequentially synthesized (or sequentially analyzed) to generate a digitized reference frame (DFd) (shown on the right side of Figure 6 and in Figure 7) containing various information. Furthermore, the original pixels (Pif) in the original image frame Fh are also digitized according to the number of frames (n) to generate reference pixels (DPkr).

[0054] Here, the terms reference frame (DFd) and reference pixel (DPkr) are used, but the reference frame (DFd) (d=1, 2, 3, . . . ) does not simply mean that image data has been synthesized, but is defined as follows.

[0055] Below, a case where one reference pixel (DPkr=11) is focused on will be described. One reference pixel (DPkr=11) contains information on the luminance standard deviation and the luminance difference determination threshold set for this luminance standard deviation. The luminance difference determination threshold is the luminance standard deviation that changes over time in response to changes in the luminance difference when the luminance value of the original pixel changes over time (when there is a luminance difference).

[0056] The luminance difference threshold is determined by taking into account the standard deviations of all other reference pixels (DPkr) (f=1, 2, 3, ... f (f = number of reference pixels in the frame)) in the reference frame (DFd=1) to which the reference pixel (DPkr=11) belongs. This is because when observing the luminance of all original pixels in the original frame, it is affected by the surrounding environment (daytime, evening, cloudy, sunny, indoors, outdoors, etc.). This influence makes the luminance difference threshold inaccurate if it is obtained using only data from multiple original pixels in the time series at one location (one area). As a result of the above measures, the "environment-adaptive luminance standard deviation" is incorporated into the "luminance difference threshold."

[0057] The following describes a technique for determining a luminance difference determination threshold value by taking into account the standard deviation of all other reference pixels (DPkr) in the reference frame (DFd) to which one reference pixel (DPkr) belongs.

[0058] First, the reference pixel (DPkr) is: It includes the average luminance value of multiple original pixels (Pif = P1f, P2f, P3f, ...) in the time series direction that make up the reference pixel (DPkr) and its standard deviation (hereafter referred to as the luminance standard deviation). This luminance standard deviation is also obtained for each reference pixel (DPkr = DP11, DP12, DP13, ...) in one reference frame (DFd) (r is the number of reference pixels in one reference frame). Therefore, the luminance difference judgment threshold (BTi) can be obtained by multiplying the luminance standard deviation per pixel in one reference frame by a certain coefficient.

[0059] If the constant coefficient is not multiplied, the luminance difference judgment threshold (BTi) will become extremely small, which can lead to the problem of detecting sensitive luminance differences such as noise. Furthermore, the luminance difference judgment threshold (BTi) may become extremely large, which can lead to the luminance difference not being detected. For this reason, it is necessary to maintain a common setting for the minimum luminance difference judgment threshold (BTi) across all frames (hereinafter referred to as the minimum judgment threshold).

[0060] Therefore, in this technology, the minimum determination threshold, that is, the determination threshold for the luminance difference (BTi) is automatically calculated according to the procedure shown in the following formula, which takes into account the luminance fluctuations of the entire reference frame (screen).

number

[0061] To prevent this provisionally determined minimum judgment threshold from becoming an extreme value (neither too small nor too large), the allowable lower limit (called the minimum threshold lower limit) and upper limit (called the minimum threshold upper limit) can be set to fixed values, and the minimum judgment threshold is made to fall within this range.

[0062] The luminance difference determination threshold (BTi) per pixel calculated above is replaced with the minimum threshold for pixels below the minimum determination threshold.

[0063] The above-mentioned luminance difference determination threshold is data used to determine the luminance fluctuation state of each of a plurality of original pixels (Pif=(i=1, 2, 3, . . . n)) in the time series direction.

[0064] 7 shows the data format of each reference pixel (DPkr=DP11, DP12, DP13, . . . DPkr) (r is the number of reference pixels in the reference frame) included in one reference frame DFd generated from original pixels for n frames. The reference pixel (DPkr) may also be referred to as information indicating the luminance state (luminance distribution characteristics) of the corresponding multiple original pixels (Pif) in the time series direction.

[0065] Specifically, the reference pixel (DPkr) is data that includes the average luminance value of the original pixel (Pif) in the time series direction that generated this reference pixel (DPkr), the luminance standard deviation of this average luminance value, the luminance difference judgment threshold (BTi), and the average value (μBTi) and its standard deviation (σBTi) for (BTi). The left side of Figure 7 shows the judgment threshold (BTi) for the brightness difference obtained by taking into account the brightness fluctuations of the entire reference frame (screen) as explained above, as well as its average value (μBTi) and standard deviation (σBTi). The deviation value diagram in Figure 7 shows the relationship between the data of the reference pixel (DPkr = DP11). There are multiple reference pixels (the number of original frame frames) in the time series direction of this reference pixel (DP11) (the direction looking down from above on a line perpendicular to the paper).

[0066] In other words, the luminance of n original pixels (n original pixels at the same position in n frames in the time series) is used to obtain the average (μ) and standard deviation (σ) of the luminance difference determination threshold (B) for the reference pixel (DP1).Then, the luminance of each of the n original pixels is used to detect luminance fluctuations.

[0067] As described above, a similar deviation value diagram can be shown for each position (DP11, DP12, DP13, . . . ) of the reference pixel (DPkr).

[0068] In an actual image capture screen, the degree of luminance dispersion (luminance dispersion value, i.e., luminance difference judgment threshold) of multiple original pixels in each time series direction of each reference pixel (DPkr=DP11, DP12, DP13, DP14, etc.) differs depending on whether the pixel is a varying pixel, a stable pixel, or an over-fluctuating pixel. In other words, the position (left-right position on the drawing) where the luminance difference judgment threshold (BTi) indicating each time series direction of each position (DP11, DP12, DP13, DP14, etc.) of the reference pixel (DPkr) is formed differs depending on the luminance dispersion value (luminance difference) in the time series direction in each area.

[0069] By utilizing this, the imaging (pixel) area in the reference frame is Pixel area (a) corresponds to a location on the image where there is no change in brightness, The pixel area corresponding to the location where the worker passes (b), For example, the pixel area (c) can be classified according to where the sheet is swaying violently due to the wind.

[0070] Below, we will explain how to classify the areas, referring to imaging positions where there is no change on the image as stable pixel areas (a), imaging positions where workers pass as variable pixel areas (b), and imaging positions where, for example, the sheet is swaying vigorously due to the wind as excessively variable pixel areas (c).

[0071] FIG. 8 shows the luminance difference decision thresholds (BTia, BTib, BTic) of reference pixels belonging to a certain stable pixel area (a), a changing pixel area (b), and an over-changing pixel area (c), generated using time-series data of a reference frame (DFd) at a certain point in time (instant). The vertical axis indicates the frequency of occurrence of the luminance difference decision thresholds of the corresponding pixels, and the horizontal axis indicates the luminance difference decision thresholds. As explained in FIG. 7, the luminance difference decision thresholds (BTia, BTib, BTic) also have average values ​​(μBTia, μBTib, μBTic) and standard deviations (σBTia, σBTib, σBTic). Of course, in this case, the luminance fluctuations of the entire reference frame (screen) are also taken into account.

[0072] Note that, here, information on stable pixels, varying pixels, and over-varying pixels (for example, the luminance difference determination threshold BTi) is explained by adding a, b, and c, such as BTia, BTib, and BTic. However, this is for ease of explanation, and the luminance difference determination threshold (BTi), average value (μBTi), standard deviation (σBTi), etc. are calculated in common for all reference pixels. Figure 8 shows how pixel variation characteristics (variation characteristics of stable pixels, varying pixels, and over-varying pixels) are classified as a result of calculating the luminance difference determination threshold (BTi), average value (μBTi), standard deviation (σBTi), etc.

[0073] In addition, in the characteristic diagrams of the determination threshold values ​​BTi for stable pixels, fluctuating pixels, and over-fluctuating pixels, the range of the characteristics of fluctuating pixels is wider than that of stable pixels and over-fluctuating pixels.

[0074] Here, we will explain a technique for determining pixel types—stable pixels, variable pixels, and over-variable pixels. First, let us assume that the average value μBTia of the luminance difference determination threshold is the average value of BTia for n original image frames, and the other values ​​μBTib and μBTic are also the average values ​​of BTib and BTic for n original image frames, respectively. Meanwhile, the luminance difference determination thresholds BTia, BTib, and BTic are specified for each original image frame. This makes it possible to observe the high / low relationship between the average values ​​μBTia, μBTib, and μBTic and the corresponding luminance difference determination thresholds BTia, BTib, and BTic.

[0075] As can be seen from the characteristic diagram in Figure 8, by first looking at the average value of the decision threshold (μBTi), the magnitude relationship between the average values ​​μBTia, μBTib, and μBTic can be determined at each point in time when the data for n original image frames has been organized. From this, it is possible to quickly determine which reference pixels are stable pixels or over-fluctuating pixels. The average value μBTia is the average value corresponding to n original image frames, and the other average values ​​of the decision thresholds μBTib and μBTic are also average values ​​corresponding to n original image frames.

[0076] Next, regarding the luminance difference determination threshold (BTib) for detecting a variable pixel, when the standard deviation (σBTib) of the determination threshold over n frames is considered, if the base is wide, that is, if the standard deviation is large, the pixel is determined to be a variable pixel. In this system, the pixel type determination threshold using the average value of the determination thresholds (μBTib) is a preset "lower threshold upper limit" that is used to determine the upper limit of the aforementioned lower threshold, and this can be used as the pixel type determination threshold (first pixel type determination threshold). Furthermore, the threshold (second pixel type determination threshold) for capturing changes in each standard deviation (σBTib) relative to the determination threshold (BTib) for brightness differences in the time series is the aforementioned lower threshold upper limit multiplied by (×1 / 2). However, these thresholds are not limited to this and can also be adjusted as sensitivity. In this case, "sensitivity may be adjusted" means that the sensitivity for determining a pixel as a variable pixel, or the sensitivity for determining a pixel as a stable pixel or an overly variable pixel may be set to a sensitive or insensitive value depending on the application environment.

[0077] For example, if the standard deviation (σBTi) of the luminance determination threshold is determined to be sufficiently large using the second pixel type determination threshold for (μBTi + σBTi), the pixel can be determined to be a variable pixel. In the case of a variable pixel, the determination threshold (BTi) changes from high to low over time.

[0078] If the pixel is determined to be other than a variable pixel based on the standard deviation of the brightness threshold (σBTi), The first pixel type determination threshold value can determine whether a pixel is stable or over-fluctuating depending on whether the average value of the luminance determination threshold value (μBTi) is large or small.

[0079] That is, pixels in the stable pixel area (a) correspond to pixels with almost no fluctuation in luminance, and because the luminance variance is low, the average luminance decision threshold (μBTia) tends to be small, and the standard deviation (σBTia) also tends to be small (so-called narrow base). On the other hand, pixels in the over-fluctuating pixel area (b) correspond to pixels where luminance fluctuations are constantly occurring, and because the luminance variance is always large, the average luminance decision threshold (μBTic) tends to be large, and the standard deviation (σBTic) tends to be small (so-called narrow base).

[0080] On the other hand, when multiple original pixel data are observed in the time series direction, pixels in the variable pixel area (b) alternate between periods of brightness fluctuation and periods of almost no fluctuation, and therefore tend to have a larger standard deviation (σBTib) (i.e., a wider base) than the stable pixel area and the over-variable pixel area.

[0081] As described above, two pixel type determination thresholds can be set: a threshold (first pixel type determination threshold) for detecting changes in the average value (μBTi) of the luminance difference determination thresholds, and a threshold (second pixel type determination threshold) for detecting changes in the standard deviation (σBTi) of the luminance difference determination thresholds.

[0082] Figure 9 shows a table for classifying stable pixels, fluctuating pixels, and over-fluctuating pixels using the average value (μBTi) and its standard deviation (σBTi) of the luminance difference judgment threshold (BTi) for each pixel, based on the characteristics of the distribution of the judgment thresholds (BTi = BTia, BTib, BTic) shown in Figures 7 and 8. Note that (BTi), (μBTi), and (σBTi) represent states where stable pixels, fluctuating pixels, and over-fluctuating pixels have not been identified.

[0083] If the average value (μBTi) and standard deviation (σBTi) of the luminance difference judgment threshold (BTi) for each pixel are less than the pixel type judgment threshold, the area of ​​pixels containing this original pixel can be determined to be an area of ​​"stable pixels." If the average value (μBTi) of the standard deviation (BTi) of the brightness difference judgment threshold fluctuates above or below the pixel type judgment threshold, and the standard deviation (σBTi) of a pixel is less than the reference threshold, the area of ​​pixels containing this original pixel can be determined to be an area of ​​``variable pixels.'' If the average value (μBTi) of the standard deviation (BTi) of the brightness difference judgment threshold is equal to or greater than the pixel type judgment threshold, and the standard deviation (σBTi) of a pixel is less than the pixel type judgment threshold, the area of ​​pixels containing this original pixel can be determined to be an area of ​​"excessively changing pixels."

[0084] Figure 10 shows the brightness difference judgment threshold (BTi) for the stable pixel (in this case, BTi is stable at a low value) (Figure 10(a)), the brightness difference judgment threshold (BTi) for the variable pixel (in this case, BTi fluctuates above and below the pixel type judgment threshold) (Figure 10(b)), and the brightness difference judgment threshold (BTi) for the over-variable pixel (in this case, BTi is stable at a high value) (Figure 10(c)).

[0085] As described above, in this embodiment, some effort is made to extract the fluctuation state of luminance. Although it is conceivable to use the standard deviation of brightness to determine the state of brightness fluctuation, in this embodiment, the judgment threshold value of the brightness difference of each pixel in the time series direction is used as the judgment data of the pixel state, which is set to prevent false detection of fluctuation pixels (pixels with brightness fluctuations) depending on the environment of the imaging area.

[0086] FIG. 11 shows a state in which the brightness fluctuation detection unit 302, the memory unit 311, and the calculation processing unit 313 shown in FIGS. 2 and 3 are associated and linked by the system controller 1000 to perform data processing and create a mixed detection composite image.

[0087] The left side of FIG. 11 shows a state in which a reference frame (DFd) having the previously described reference pixels (DPk) in two-dimensional directions has been constructed.

[0088] In one reference frame (DFd), determination result data (which may be called detection data) that detects whether each original pixel present in the time series direction is a stable pixel, an over-fluctuating pixel, or a fluctuating pixel is also obtained in the time series direction (see Figure 10). Here, when a set of original pixel data is obtained, a second mask process is performed.

[0089] The judgment result data is shown as a brightness level, with the judgment condition being that the conditions shown in Fig. 9 are satisfied. For example, an over-varying pixel is given a brightness level of 225, a stable pixel a brightness level of 0, and a varying pixel a brightness level of 110. However, these brightness levels are not limited to these, and many other discrimination methods can be set, such as discrimination by color or pattern, rather than discrimination by brightness level.

[0090] Next, a display frame is prepared. Then, the above-mentioned determination result data (data indicating whether the original pixel data is a stable pixel, an excessively variable pixel, or a variable pixel) is written at each pixel position in the display frame. The type of the determination result data, i.e., the number of bits and value, is not particularly limited; the key point is that it is sufficient as long as it is detection data that can identify stable pixels, excessively variable pixels, and variable pixels.

[0091] The n display frames, each with determination result data written at the pixel position, are combined in a combining calculator 1500 to form a single display frame. However, in this case, when viewed in chronological order, weights are assigned to the determination result data in increasing order from the past to the present. This is because the most recent determination result data is given more importance. In this example, the weights are assigned in the order 1, 4, 9, 16, and 25, starting from the oldest determination result data, but the weights are not necessarily limited to these values ​​or patterns. As the memory used for adding or discarding the data, for example, a circular cache memory is used to repeatedly discard and add data.

[0092] The display data (indicating brightness levels) output from the synthesis calculator 1500 is multiplied by each weight by each value of the frame data, and the result is divided by the sum of the weights (55 in this case) in a divider 1501 to correct the value to a range suitable for display and input it to the display unit 2000. An image appears on the display unit 2000, which shows the areas where variable pixels exist.

[0093] 12 shows a first brightness fluctuation detection result image 2021 in which varying pixels indicate a detection area within the input image data 20, and a mask image 2050 indicating a preset mask area. The first brightness fluctuation detection result image 2021 shows an image of the work area obtained by a person working (the shaded area on the right side of 2021) and an image of the area where a forklift was active (the shaded area on the left side of 2021).

[0094] Furthermore, information on the mask area (the area outside the blank (white) portion in the center) of the mask image 2050 is used to perform mask processing on the brightness fluctuation detection result image 2021, resulting in a second brightness fluctuation detection result image 2022. In this mask processing, although one second brightness fluctuation detection result image 2022 described above is shown here, in reality, at this processing stage, there are multiple brightness fluctuation result images with different weightings for synthesis, as described in FIG. 11. In other words, this image is the brightness fluctuation detection composite image 2023, and this image is input to the display device 2000. It is also input to an image classification unit 2024. The image classification unit 2024 can perform image classification on the brightness fluctuation detection composite image 2023 using, for example, a neural network, and determine and output the activity state.

[0095] According to the above-described embodiment, in the brightness fluctuation detection operation, it is possible to determine brightness fluctuations without being affected by noise (such as other sudden fluctuations in brightness on the image). Conventional technology focuses only on brightness fluctuations and performs movement state determination using information reduced to a low dimension, which can result in erroneous determination due to the significant influence of brightness fluctuations such as noise. However, according to this embodiment, the state of brightness fluctuations is visualized, and the level of information is increased so that the movement (activity) state of a moving object can be determined using image recognition such as AI. As a result, the influence of noise is reduced in the determination of brightness fluctuations, making it possible to improve the accuracy of movement state determination.

[0096] 13 shows an example of a synthesis processing unit for the first brightness fluctuation detection result image 2021 and a mask image 2050, and these two images are subjected to logical operation processing by an AND circuit 2051. This processing results in a second brightness fluctuation detection result image 2022.

[0097] Fig. 14 shows how multiple pieces of judgment result data (five frames in this case) are combined into a single display frame (composite image) by the combination calculator 1500 and divider 1501 shown in Fig. 11. The blank (white) areas in the figure indicate areas where no work is performed as stable pixels, and the marked areas indicate traces of work by workers (or moving objects). The figure also shows the calculation formula used for performing weighted averaging. In the example shown, W is the output of the divider 1501, n is 5, wi is the weight, and Xi is the judgment result data.

[0098] Figure 15 is a block diagram showing another embodiment from the perspective of software configuration. Images captured by the 360-degree camera 10 are input to the stream receiving unit SW1. The stream receiving unit SW1 supplies the image data to the recorded video processing unit SW2 and also to the capture unit SW3. The recorded video processing unit SW2 sequentially records the image data in memory. It is also possible to send the recorded image data to the capture unit SW3 in response to a request from the capture unit SW3.

[0099] The capture unit SW3 dewarps the input image data and supplies the dewarped image data to the clipping / correction processing unit SW4. The clipping / correction processing unit SW4 corresponds to the pre-processing unit 301 in Figures 1 and 2. The pre-processing unit 301 clips out image data of an area suitable for grasping the activity state and supplies the clipped image data to the brightness fluctuation detection unit SW5.

[0100] The brightness fluctuation detection unit SW5 corresponds to the brightness fluctuation detection unit 302 shown in Figures 1 and 2. It generates a data frame by combining original image frames in a time series spanning multiple frames, and executes data processing up to a stage equivalent to the processing of the combination calculator 1500 described in Figure 14.

[0101] Next, the masking processor SW6 performs the masking process described in Figures 12 and 13. The activity state determination processor SW7 uses the masked image data to obtain the final composite image described in Figure 18, and obtains the activity state determination result by image classification. This composite image and the activity state determination result are further input to the operating status measurement processor SW8.

[0102] The operational status measurement processing unit SW8 supplies the final composite image described in FIG. 14 to the display processing unit SW9. The display processing unit SW9 edits the composite image as a display image to be displayed on the display HW01. The display image can take various forms. For example, it is also possible to provide judgment information such as whether the operational status (work status) of a moving object (e.g., a worker) is normal or abnormal. Note that judgment criteria can be provided to the operational status measurement processing unit SW8 in advance. For example, as shown in Figure 14, if the width, length, etc. of the "marker part (work area)" is within the range of the pre-calculated width, length, etc., it is judged as "normal," and if it deviates from the pre-calculated width, length, etc., it is judged as abnormal. If it is judged as "abnormal," it is also possible to issue an alarm signal to the alarm device HW02.

[0103] Furthermore, the operating status measurement processing unit SW8 can detect the display state of the "marker portion (work area)" shown in FIG. 14 and the "duration" of the activity determination result, as well as the time period. If the detected "duration" is the "scheduled time," the work is proceeding smoothly. If it is the "irregular time," the progress of the work is disrupted, and the cause can be investigated. Furthermore, by monitoring the "duration" and "rest time," it can also be used to determine whether breaks are being taken normally.

[0104] The data measured and detected by the operational status measurement processing unit SW8 is recorded in memory by the operation log processing unit SW10 according to type.

[0105] A data accumulation processing unit SW21 is also provided, which receives as input the data frames obtained by the brightness fluctuation detection unit SW5, the brightness fluctuation detection result image data obtained by the mask processing unit SW6, and the brightness fluctuation detection composite image data from the activity state determination processing unit SW7. The data accumulation processing unit SW21 can organize the various input data by type and time axis and accumulate them in the processing result log processing unit SW22.

[0106] The data recorded here can be used later to check the system operation, and can also be used later to verify system or on-site accidents.

[0107] Therefore, the data accumulation processing unit SW21 and the processing result log processing unit SW22 may be located remotely, for example via communication equipment (hardware), so as to be isolated from the brightness fluctuation detection unit SW5, the mask processing unit SW6, and the activity status determination processing unit SW7.

[0108] FIG. 16 is a block diagram showing yet another embodiment from the viewpoint of hardware configuration.

[0109] An image captured by the 360-degree camera 10 is input to the image buffer memory HW2 via the input terminal HW1. At this time, the image is, for example, (2992 pixels x 2992 pixels). Various preset data is provided to the input terminal HW3 and stored in the preset storage unit HW4. The preset information Ps1, Ps2, ... Psn includes preset information for setting an area for setting the cutout position, an area for performing first mask processing, an area for performing second mask processing, and also for setting the system operation mode.

[0110] The output data of the buffer memory HW2 is input to the cropped image creation unit HW5. The cropped image creation unit HW5 dewarps the area-of-interest image data using preset information, crops the dewarped image data, and stores the cropped image data in the cropped image buffer memory HW6. At this stage, the cropped image data is, for example, (640 pixels x 480 pixels). This image data is input to the image processing unit HW7.

[0111] The image processing unit HW7 includes an image processing unit HW7, a brightness fluctuation detection unit HW71, a mask processing unit HW72, and a state determination processing unit HW73.

[0112] These correspond to the brightness fluctuation detection unit SW5, mask processing unit SW6, and activity state determination processing unit SW7 shown in Fig. 15. The brightness fluctuation detection unit HW5 generates a data frame by combining multiple time-series original image frames, and performs data processing up to the stage corresponding to the processing by the combination calculator 1500 described in Fig. 11. The mask processing unit HW72 performs the mask processing described in Figs. 12 and 13. The activity state determination processing unit HW73 then uses the masked image data to obtain the final combined image described in Fig. 14.

[0113] The data resulting from the above processing (such as data frames, output data from the synthesis calculator 1500, and masked image data) is stored sequentially in a processing result buffer memory HW8.

[0114] The composite image data for display is input to the display processing unit HW9 and displayed on the front display device HW10.

[0115] The composite image is also input to the operation status measurement processing unit HW11 via the processing result buffer memory HW8. The operation status measurement processing unit HW11 can determine an abnormal state from the movement trajectory of the moving object (worker), and if an abnormality is detected, it sends a warning signal to the warning device HW2.

[0116] In the hardware configuration, a memory for storing an operation log and a processing result log memory for storing a processing result log may also be provided.

[0117] The above system may be provided with the following functions. For example, in the preprocessing unit, a color signal filter of a specific color may be used for a color image to detect only moving objects of a specific color. Furthermore, a process of masking a background image of a specific color may be performed using a color signal filter of a specific color, or this may be combined with the above-mentioned process of detecting moving objects of a specific color. Furthermore, these processes may be arbitrarily switchable.

[0118] Furthermore, although the above embodiment has been described as a system for monitoring truck berths at logistics bases, it can also be used as a system for monitoring changes in materials, changes in the amount of soil and sand, and work conditions at construction sites, civil engineering sites, etc.

[0119] Next, the characteristic aspects of the above-described embodiment will be summarized below. (A1) (a11) An activity status determination system having a data processing unit including a luminance variation detection means and an imaging means (302, AS3, AS4), (a12) A captured frame is defined as an original frame (Fh) including a plurality of original pixels (Pif) in different areas within the frame; (a13) generating reference pixels (DPkr) having information obtained by sequentially analyzing each of the plurality of original pixels (Pif) in the original image frame (Fh) per a predetermined number of frames (n) in a time series direction, and a frame in which the reference pixels (DPkr) are two-dimensionally arranged is defined as a reference frame (DFd); (a14) When one of the reference pixels (DPkr) is focused on, The original pixel (Pif) includes a determination threshold value for a luminance difference in a time series direction, (a15) The brightness fluctuation detection means of the data processing unit is a means for obtaining determination result data indicating an area of ​​reference pixels (DPkr) corresponding to the change in level as an area of ​​changed pixels when the determination threshold value of the luminance difference changes in the time series direction; (a16) The imaging means of the data processing unit includes: The device includes a means for converting each judgment result data detected at the reference pixels (DPkr) in different areas of the plurality of reference frames in the time series direction into display pixels in an area of ​​one display frame that corresponds to the different areas.

[0120] (a17) The image identification means of the data processing unit is The apparatus includes a means for determining and classifying the state obtained from the image and extracting information using the data imaged by the imaging means.

[0121] (A2) In A1, when one reference pixel (DPkr) is focused on, The threshold for determining the brightness difference in the time series direction is calculated for each frame based on the average brightness value and brightness standard deviation of each original pixel (Pif) in the time series direction, and the average brightness standard deviation and other brightness standard deviations of other reference pixels (DPkr) in the frame.

[0122] (A3) In A2, the average value and standard deviation for n frames in the time series direction are obtained for the luminance difference determination threshold, and these average value and standard deviation are used as "data to be used for pixel type determination."

[0123] (A4) In A1, the imaging means is A preset area in the display frame is masked to prevent unnecessary judgment result data from being converted into display pixels.

[0124] (A5) In A1, the imaging means is When performing imaging processing to convert the area of ​​the variable pixels into display pixels that fall within a predetermined brightness range, the apparatus has a means for combining and averaging judgment result data of multiple judgment result data of multiple reference pixels in multiple reference frames in the time series direction, and converting the result data into the display pixels in one frame.

[0125] (A6) In A1, the imaging means performs the averaging process as a weighted averaging process, and the weight of the determination result data at a newer time point in the time series direction is greater than the weight of the determination result data at an older time point.

[0126] (A7) In A1, the original pixel (Pif) from which one reference pixel (DPkr) is generated is a preset color signal component.

[0127] (A8) An activity status determination device including the system of A1, or an activity status determination method in which the system of A1 operates, or a service providing device including a processing result storage that accumulates processing result data and a communication unit, and capable of transmitting the processing result data to an external device, and and a determination program for executing the activity status determination method.

[0128] In the above description, with regard to terms such as generating means, constructing means, and processing unit, it goes without saying that the scope of the present invention can be achieved even if the terms "means" and "unit" are replaced with "device," "unit," "block," and "module." Furthermore, with regard to each component of the claims, even if the component is expressed separately, or if multiple components are expressed together, or if these are expressed in combination, it also falls within the scope of the present invention. Furthermore, even if a claim is expressed as a method, it is still an application of the device of the present invention.

[0129] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0130] 10...360-degree camera, 11...image data, 20...input image data, 100...capture unit, 300: Activity state determination unit, 301: Preprocessing unit, 302: Brightness variation detection unit, 303: Image identification unit, 311: Memory, 313: Arithmetic processing unit, 401: Preprocessing unit, 402: Object detection unit, 500: Peripheral information processing unit, 511: Memory, 513: Arithmetic processing unit, 601: Preprocessing unit, 602: Region extraction unit, 700···Post-processing unit, 900···User interface, 1000···System controller, Px···Original pixel, DPx···Reference pixel, Fn···Original frame, DFn···Reference frame.

Claims

1. An activity status determination system having a data processing unit including a luminance variation detection means and an imaging means, A captured frame is defined as an original frame including a plurality of original pixels in different areas within the frame, a plurality of original pixels in the original image frame are sequentially analyzed in a time series direction for a predetermined number of frames to generate reference pixels having information, and a set of the reference pixels is defined as a reference frame; When one of the reference pixels is considered, a determination threshold value for a luminance difference in the luminance of the original pixel in a time series direction, The brightness fluctuation detection means a means for obtaining a determination result indicating an area of ​​reference pixels corresponding to the change in level as an area of ​​changed pixels when the determination threshold value of the luminance difference changes in the time series direction; The imaging means a means for converting each determination result data detected at the reference pixels in different areas of the plurality of reference frames in the time series direction into display pixels in an area of ​​one display frame that corresponds to the different areas; Activity status determination system.

2. When one of the reference pixels is considered, 2. The activity status determination system of claim 1, wherein the threshold for determining the brightness difference in the time series direction is calculated for each frame based on the average brightness value of each original pixel in the time series direction and the brightness standard deviation relative to this average brightness value, and the average brightness standard deviation and other brightness standard deviations of other reference pixels in the frame.

3. 3. The activity status determination system according to claim 2, wherein the average value and standard deviation in the time series direction are calculated for the luminance difference determination threshold, and the pixel type is determined based on the relationship between the average value or standard deviation in the time series direction and the luminance difference determination threshold.

4. The imaging means 2. The activity status determination system according to claim 1, wherein a preset area in said display frame is masked to prevent unnecessary determination result data from being converted into display pixels.

5. The imaging means 2. The activity status determination system according to claim 1, further comprising means for synthesizing and averaging a plurality of determination result data of a plurality of reference pixels in a plurality of reference frames in a time series direction when performing imaging processing to convert the area of ​​the variable pixels into display pixels that fall within a predetermined brightness range, and converting the result data into the display pixels in one frame.

6. The imaging means 6. The activity status determination system according to claim 5, wherein the averaging process is a weighted averaging process, and the weight of the determination result data at a later time point in the time series is greater than the weight of the determination result data at an earlier time point.

7. an original pixel from which one reference pixel is generated is a predetermined color signal component; The activity status determination system according to claim 1 .

8. An activity status determination device comprising the activity status determination system according to claim 1.

9. The system according to claim 1 further comprises a processing result storage that accumulates the determination result data, and a communication unit; A service providing system that is capable of transmitting the determination result data to an external device.

Images