Image processing device, image processing system, image processing method, and program

Abstract

The present disclosure provides a video processing device, a video processing system, and a video processing method that are capable of suppressing the influence of changes occurring in image quality of a video and improving the accuracy of video recognition. A video processing device (10) according to one aspect of the present disclosure comprises a characteristic information generation unit (11) that generates image quality characteristic information indicating spatio-temporal characteristics of image quality information indicating an image quality of a video, an integration unit (12) that produces integrated data by integrating video information including spatio-temporal characteristics of the video with the image quality characteristic information generated by the characteristic information generation unit (11), and a recognition unit (13) that executes recognition processing on a subject shown in the video on the basis of the integrated data.

Description

Video processing device, video processing system, and video processing method

[0001] The present disclosure relates to a video processing device, a video processing system, and a video processing method.

[0002] Video processing technology has been advancing in recent years.

[0003] For example, Patent Literature 1 discloses a method for identifying a predetermined object from image data that may include the object in a cloud server. Specifically, when encoding video data including image data, the cloud server generates encoding parameter features, which are features regarding mapping information in which encoding parameters determined for each unit image area are mapped to the unit image area, and image features, which are features related to pixel values ​​of the image data. The cloud server then inputs the generated encoding parameter features and image features into a trained classifier, which then outputs information related to the class of the predetermined object, thereby identifying the object from the image data.

[0004] Patent Literature 2 also discloses a moving image processing device that performs quantization processing of a face region such that the reduction in the compression rate for the face region is small when the area ratio of the face region to the entire input image is relatively large, and the reduction in the compression rate for the face region is large when the area ratio of the face region to the entire input image is relatively small.

[0005] JP 2021-043773 A JP 2010-193441 A

[0006] If the image quality of the image used in the recognition process changes over time, the recognition engine may not be able to accurately recognize the changed image. The technology disclosed in Patent Document 1 aims to reduce the processing load by using "encoding parameter features" in the recognition process, but does not solve this problem. Furthermore, the technology disclosed in Patent Document 2, which balances the compression rate between the face area and other areas, also does not solve this problem.

[0007] An object of the present disclosure is to provide an image processing device, an image processing system, and an image processing method that can suppress the impact of changes in image quality in an image and improve the accuracy of image recognition.

[0008] An image processing device according to one aspect of this embodiment includes a feature information generation unit that generates image quality feature information that indicates the spatiotemporal features of image quality information that indicates the image quality of an image, an integration unit that generates integrated data that integrates information about the image that includes the spatiotemporal features of the image and the image quality feature information generated by the feature information generation unit, and a recognition unit that performs recognition processing on a subject included in the image based on the integrated data.

[0009] An image processing system according to one aspect of this embodiment includes a feature information generation unit that generates image quality feature information that indicates the spatiotemporal features of image quality information that indicates the image quality of an image, an integration unit that generates integrated data that integrates information about the image that includes the spatiotemporal features of the image and the image quality feature information generated by the feature information generation unit, and a recognition unit that performs recognition processing on a subject included in the image based on the integrated data.

[0010] One aspect of the image processing method according to this embodiment is executed by a computer, and generates image quality feature information indicating the spatiotemporal characteristics of image quality information that indicates the image quality of the image, generates integrated data that integrates information about the image that includes the spatiotemporal characteristics of the image with the image quality feature information, and performs recognition processing on the subject included in the image based on the integrated data.

[0011] According to the present disclosure, it is possible to provide an image processing device, an image processing system, and an image processing method that can suppress the impact of changes in image quality and improve the accuracy of image recognition even when changes occur in the image quality.

[0012] FIG. 1 is a block diagram showing an example of a video processing device according to a first embodiment. FIG. 2 is a flowchart showing an example of representative processing of the video processing device according to the first embodiment. FIG. 3 is a block diagram showing an example of a video processing system according to the first embodiment. FIG. 4 is a block diagram showing an example of a video recognition system according to a second embodiment. FIG. 5 is a block diagram showing an example of a center server according to the second embodiment. FIG. 6 is a block diagram showing an example of a compression information integrating unit according to the second embodiment. FIG. 7 is a diagram showing an example of QP map information. FIG. 8 is a diagram showing an example of generated attention map information. FIG. 9 is a flowchart showing an example of representative processing of the center server according to the second embodiment. FIG. 10 is a block diagram showing another example of a compression information integrating unit according to the second embodiment. FIG. 11 is a block diagram showing an example of a hardware configuration of the device according to each embodiment.

[0013] Hereinafter, each embodiment will be described with reference to the drawings. Note that the following description and drawings have been omitted or simplified as appropriate for clarity of explanation.

[0014] First Embodiment (1A) Hereinafter, a first embodiment of the present disclosure will be described with reference to the drawings. In this (1A), a video processing device will be described.

[0015] 1 is a block diagram showing an example of a video processing device. The video processing device 10 includes a feature information generating unit 11, an integrating unit 12, and a recognizing unit 13. Each unit (means) of the video processing device 10 is controlled by a control unit (controller) not shown. Each unit will be described below.

[0016] [Configuration Description] The feature information generation unit 11 generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of a video. The video is data on which recognition processing regarding a subject is performed, and is assumed to be, for example, data acquired by a camera or the like, but is not limited to this. The video is data that includes multiple still images (hereinafter simply referred to as images) in a time series. Note that in the present disclosure, the terms video and image are interchangeable. That is, the video processing device 10 can be referred to as a video processing device that processes video, or as an image processing device that processes images. The video processing device 10 can acquire this video, for example, from outside the video processing device.

[0017] The image quality information is any information indicating image quality, and may be, for example, information indicating the degree of compression of a region of a frame (image frame) included in a video, or lightness information or luminance information of the video, etc. An example of the information indicating the degree of compression of a region of a frame included in a video is, but is not limited to, a QP (Quantization Parameter) map, which is a map of spatiotemporal features of image quality information.

[0018] The integration unit 12 generates integrated data by integrating information about the video, including the spatiotemporal features of the video, with the image quality feature information generated by the feature information generation unit 11. The information about the video may be information indicating the spatiotemporal features of the video after any processing has been performed on the video (video feature information), or the video itself. More specifically, the video feature information is a feature amount related to pixel values ​​of the video, and can be expressed, for example, by a matrix indicating the feature amount. The video feature information may be generated by the video processing device 10 based on the video, or may be generated by a device external to the video processing device 10.

[0019] Furthermore, the integrating unit 12 can use any method for integrating the data as long as the integrated data reflects image quality feature information on the information related to the video. For example, the integration may be performed by any arithmetic process such as multiplication or addition, by an algorithm based on a predefined rule base, or by a pre-trained AI (Artificial Intelligence) model such as a neural network. Details of this will be described later in the second embodiment.

[0020] The recognition unit 13 performs recognition processing on the subject included in the video based on the integrated data generated by the integration unit 12. The recognition unit 13 is capable of performing any recognition processing on the subject, and may, for example, identify the attributes of the subject. The attributes of the subject may indicate the type of object defined for the subject, such as whether the subject is a human, a non-human organism, or a machine such as a bicycle, automobile, or robot. Furthermore, if the subject is a human, the attributes of the subject may be information that can uniquely identify the subject, such as whether the subject is one of persons A, B, C, etc. pre-stored in the video processing device 10, or whether the subject is an unknown person who is not stored. Furthermore, if the subject is a human, the attributes of the subject may be information that identifies the occupation of the subject (for example, whether the subject is a construction worker, a plasterer, or an ordinary passerby). If the subject is a machine, the attributes of the subject may be information that identifies the type of machine, such as whether the subject is a bicycle, automobile, or industrial robot. As another example, the recognition unit 13 may identify the movement the subject is performing. The movement of the subject is, for example, the behavior of the subject when the recognition unit 13 identifies the subject as a human, and the work content of the subject when the recognition unit 13 identifies the subject as a robot.

[0021] The recognition unit 13 may be, for example, an AI model (e.g., a neural network) that has been trained in advance. The training is performed by inputting sample videos including a subject and training data including a correct answer label indicating what the subject is or what movement the subject is making for each video into the recognition unit 13 (or the video processing device 10). Alternatively, the recognition unit 13 may analyze the video based on a predefined rule base to determine what the subject is or what movement the subject is making.

[0022] 2 is a flowchart showing an example of a typical process of the video processing device 10, and this flowchart will be used to explain an overview of the process of the video processing device 10. Note that the details of each process are as described above, and therefore will not be explained again.

[0023] First, the feature information generator 11 generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of the video (step S11; generation step). The integrator 12 generates integrated data by integrating information about the video and the image quality feature information generated by the feature information generator (step S12; integration step). The recognizer 13 performs recognition processing on the subject included in the video based on the integrated data (step S13; recognition step).

[0024] [Explanation of Effects] As described above, the recognition unit 13 can perform recognition processing on a subject based on integrated data on the video that reflects image quality feature information. In other words, even if a change in image quality occurs in the video, the recognition unit 13 can perform recognition processing after grasping the information as image quality feature information. Therefore, the influence of the change in image quality that occurs in the video can be suppressed, and the accuracy of video recognition can be improved.

[0025] (1B) Next, in (1B), a video processing system will be described. Fig. 3 is a block diagram showing an example of a video processing system. The video processing system 20 includes a feature information generation device 21 and a recognition device 22. The feature information generation device 21 has a feature information generation unit 11, and the recognition device 22 has an integration unit 12 and a recognition unit 13. The feature information generation unit 11 to the recognition unit 13 perform the same processing as that shown in (1A). When the feature information generation unit 11 generates image quality feature information, the generated image quality feature information is output to the recognition device 22. The integration unit 12 uses the image quality feature information to perform the processing shown in (1A).

[0026] As described above, the video processing according to the present disclosure may be implemented by a single device, as shown in (1A), or may be implemented as a system in which the processing is distributed across multiple devices, as shown in (1B). Note that the device configuration shown in (1B) is merely exemplary. As another example, a first device may have a feature information generation unit 11 and an integration unit 12, and a second device may have a recognition unit 13. Alternatively, three different devices may be provided, each having a feature information generation unit 11, an integration unit 12, and a recognition unit 13. As yet another example, the video processing system 20 may be partially or entirely provided on a cloud server built on a cloud, or on another type of virtualized server created using virtualization technology, etc. Functions other than those provided on such servers are located at the edge. For example, in a system that monitors video captured on-site via a network, the edge is a device located at or near the site, and is also a device close to the terminal in the network hierarchy.

[0027] In the following embodiment 2, a specific example of the video processing device 10 described in embodiment 1 will be disclosed. However, the specific example of the video processing device 10 described in embodiment 1 is not limited to the one shown below. Furthermore, the configuration and processing described below are examples and are not limited to these.

[0028] (2A) [Configuration Description] Fig. 4 is a block diagram showing an example of a video recognition system. The video recognition system 100 includes a terminal 101, a base station 102, a multi-access edge computing (MEC) server 103, and a center server 104. In the example of Fig. 4, the terminal 101 is provided on the edge side (field side) of the video recognition system 100, and the center server 104 is located at a location away from the field (cloud side). Each device will be described below.

[0029] Terminals 101A, 101B, and 101C (hereinafter collectively referred to as terminals 101) are edge devices connected to a network, have cameras as image capture units, and can capture images of any location. Terminal 101 transmits the captured images to center server 104 via base station 102. In this example, terminal 101 transmits the images via a wireless line. However, the images may also be transmitted via a wired line.

[0030] However, the terminal 101 and the camera may be provided separately. In this case, the camera transmits the captured video to the terminal 101, which is a relay device, and the terminal 101 processes the video as necessary and transmits it to the center server 104 via the base station 102. However, the camera may process the video and transmit it to the terminal 101, and the terminal 101 may then transmit the video.

[0031] Furthermore, as will be described later, each terminal 101 is assigned a bit rate for video that can be transmitted to the center server 104 by the MEC server 103. The video bit rate means the amount of video data per unit time (e.g., one second). The assigned bit rate can change over time. Each terminal 101 can reduce (i.e., compress) the bit rate of a part or all of the captured video by a predetermined percentage so that the bit rate of the video transmitted to the center server 104 is equal to or lower than the assigned bit rate.

[0032] Furthermore, when the terminal 101 detects that a predetermined condition is satisfied, it can reduce the bit rate of a partial region or the entire region of a frame of the captured video by a predetermined percentage. The terminal 101 may perform this process by, for example, analyzing the captured video. In particular, when the terminal 101 detects that a predetermined object (e.g., a predetermined person) is included in a frame of the captured video, it can reduce the bit rate of regions other than that region by a predetermined percentage compared to the bit rate of that region. However, the terminal 101 can also reduce the bit rate of regions including the predetermined object by a predetermined percentage compared to the bit rate of regions other than that region. As another example, when the terminal 101 detects that it is in a predetermined environment (e.g., when the video is captured during a predetermined time period), it can reduce the bit rate of the entire frame of the captured video by a predetermined percentage.

[0033] In this way, when the terminal 101 performs video compression under predetermined conditions, the terminal 101 generates QP map information, which is information indicating the degree of compression of a frame region included in the video, and transmits the QP map information to the base station 102. The terminal 101 may also uniformly compress the video to be transmitted so that it can be decompressed later by the center server 104.

[0034] The base station 102 transfers video transmitted from each terminal 101 to the center server 104 via the network. The base station 102 also transfers control signals from the MEC server 103 to each terminal 101. For example, the base station 102 is a local 5G (5th Generation) base station, a 5G gNB (next generation Node B), an LTE eNB (evolved Node B), a wireless LAN access point, or the like, but may also be another relay device. The network is, for example, a core network such as a 5GC (5th Generation Core network) or an EPC (Evolved Packet Core), the Internet, or the like.

[0035] The MEC server 103 allocates a bit rate for the video transmitted from each terminal 101 to the base station 102, and transmits information about the allocated video bit rate as control information to each terminal 101. Each terminal 101 adjusts the video bit rate in accordance with the control information, as described above. Note that the base station 102 and the MEC server 103 are connected to each other so that they can communicate with each other using any communication method, but the base station 102 and the MEC server 103 may also be integrated into a single device.

[0036] The MEC server 103 detects at least one of the communication environment between each terminal 101 and the base station 102 or the communication environment between the base station 102 and the MEC server 103, and based on the detection result, determines the video bit rate to be allocated to each terminal 101. At this time, the MEC server 103 predicts the accuracy with which the center server 104 (described below) will recognize an object based on the video captured by each terminal 101, and can determine the video bit rate to be allocated to each terminal 101 so that the predicted recognition accuracy for the video captured by each terminal 101 is maximized in total.

[0037] The MEC server 103 transmits information about the determined bit rate as control information to each terminal 101. Each terminal 101 adjusts the bit rate of the video to be transmitted to the center server 104 based on this control information.

[0038] The communication environment between each terminal 101 and the base station 102 may be determined by, for example, at least one of the number of terminals 101, the degree of congestion in wireless communication between each terminal 101 and the base station 102, and the quality of wireless communication. An example of the degree of congestion in wireless communication is the number of packets per unit time, and an example of the quality of wireless communication is radio wave strength (RSSI: Received Signal Strength Indicator), but these are not limited to these. The communication environment between the base station 102 and the MEC server 103 may be determined by, for example, at least one of the degree of congestion in wireless communication between the base station 102 and the MEC server 103 and the quality of wireless communication. The MEC server 103 can detect at least one of the communication environment between each terminal 101 and the base station 102 or the communication environment between the base station 102 and the MEC server 103 using one or more of the parameters described above.

[0039] Furthermore, the MEC server 103 may set predetermined conditions for reducing the bit rate of a part or all of the area of ​​the video captured by the terminal 101, and transmit the setting information to each terminal 101. When the terminal 101 detects that the predetermined conditions have been satisfied based on the setting information, it can reduce the bit rate of a part or all of the area of ​​the video captured.

[0040] As described above, the video recognition system 100 can reduce the bit rate of video transmitted from the terminal 101 in certain cases. This makes it possible to reduce the processing load when processing is performed on the center server 104 side and the communication load within the system. However, due to fluctuations in network communication quality, there is a possibility that video from the terminal 101 will not be transmitted with high quality or accurately. Furthermore, when video, which is time-series data, is transmitted from the terminal 101, block noise may occur due to fluctuations in communication quality, etc. For these reasons, if changes in the image quality of the video occur, there is a possibility that the recognition accuracy of the video will decrease when the video is analyzed. However, the center server 104 described below can suppress such phenomena.

[0041] 5A is a block diagram showing an example of a center server. The center server 104 includes a video acquisition unit 111, a QP map information acquisition unit 112, a compression information integration unit 113, and a behavior recognition unit 114. The center server 104 executes the following video processing for each terminal 101. Each unit of the center server 104 will be described below.

[0042] The video acquisition unit 111 is an interface that acquires video transmitted from each terminal 101 via the base station 102 and QP map information corresponding to the video. As described in the first embodiment, the QP map information is information indicating the degree of compression of a frame region included in the video. Note that, if the video transmitted from each terminal 101 is uniformly compressed, the video acquisition unit 111 executes a decompression process to enable the recognition process described below. The video acquisition unit 111 outputs the acquired information to the QP map information acquisition unit 112 and the compression information integration unit 113.

[0043] QP map information acquisition unit 112 extracts and acquires QP map information indicating the degree of compression of the bit rate of the video from the information acquired from video acquisition unit 111. Note that if QP map information is not transmitted from terminal 101, QP map information acquisition unit 112 can acquire QP map information corresponding to the video by analyzing the video output from video acquisition unit 111. QP map information acquisition unit 112 outputs the acquired QP map information to compression information integration unit 113.

[0044] The compressed information integration unit 113 generates integrated data for each frame of the video by integrating the video and the image quality feature information created based on the QP map information, and outputs the data to the behavior recognition unit 114. Details of this will be described later.

[0045] The behavior recognition unit 114 corresponds to the recognition unit 13 according to the first embodiment, and recognizes the behavior of a person who is a subject of the video by analyzing the integrated data output from the compressed information integration unit 113. The behavior recognition unit 114 may be an AI model (e.g., a neural network) that has been trained in advance. The learning method is similar to that of the recognition unit 13, and therefore a description thereof will be omitted. Alternatively, the behavior recognition unit 114 may determine the movement of the subject by analyzing the video based on a predefined rule base.

[0046] 5B is a block diagram showing an example of the compressed information integration unit 113. The compressed information integration unit 113 has a feature information generation unit 120 having an attention map generation unit 121, and a feature integration unit 122. Each unit of the compressed information integration unit 113 will be described below.

[0047] The feature information generation unit 120 corresponds to the feature information generation unit 11 according to the first embodiment. The attention map generation unit 121 included in the feature information generation unit 120 uses the QP map information output from the QP map information acquisition unit 112 to generate, for each frame of video, attention map information indicating an area within the frame that should be focused on in recognition processing (hereinafter also referred to as an attention area). The attention map information is a map of spatiotemporal feature amounts of the QP map information. Hereinafter, with reference to Figures 6A and 6B, an example of how the attention map generation unit 121 generates attention map information will be described.

[0048] 6A is a diagram showing an example of QP map information, showing QP map information (QP map sequence) for each frame in a time series of times T=t1, t2, t3, .... F1 to F3 in the QP map at each time indicate the area of ​​the entire frame. Therefore, the QP map information indicates spatiotemporal information.

[0049] In FIG. 6A , hatched areas H1 and H2 in frame F2 are areas where the degree of compression is greater than that of the other areas in frame F2. For example, it is assumed that terminal 101 performs processing to reduce the video bit rate for hatched areas H1 and H2, but does not perform processing to reduce the video bit rate for the other areas. Alternatively, terminal 101 may perform processing to significantly reduce the video bit rate for hatched areas H1 and H2, but perform processing to reduce the degree of bit rate reduction for the other areas compared to hatched areas H1 and H2. Similarly, hatched area H3 in frame F3 is also an area where the degree of compression is greater than that of the other areas in frame F3. In this way, the QP map sequence indicates the degree of compression of the video bit rate in time and space.

[0050] In addition, the positions and sizes of regions with high and low compression levels in the QP map sequence change over time. For example, at a certain time, there may be regions with high compression levels in the entire frame, at another time, there may be regions with low compression levels in the entire frame, and at yet another time, there may be a mixture of regions with high and low compression levels in the frame.

[0051] The hatched areas H1 to H3 have a reduced video bit rate, so it is considered difficult to perform accurate recognition processing (inference processing) on ​​those areas even if the video of those areas is input to the behavior recognition unit 114. Furthermore, including such areas in the recognition processing would increase the processing load on the center server 104.

[0052] The attention map generation unit 121 determines whether there is a region in the QP map for each time shown in FIG. 6A where the bit rate has decreased from the reference value by a predetermined threshold or more. If there is a region where the degree of bit rate decrease is equal to or greater than the predetermined threshold, the attention map generation unit 121 excludes that region from the region of interest. In other words, the attention map generation unit 121 masks that region. On the other hand, if there is a region where the degree of bit rate decrease is less than the predetermined threshold, the attention map generation unit 121 leaves that region as a region of interest (i.e., a region valid for inference processing). Note that information on the reference value and threshold used for the determination is stored in a storage unit (not shown) in the center server 104, and the attention map generation unit 121 acquires that information when performing this determination.

[0053] FIG. 6B is a diagram showing an example of attention map information generated by the attention map generation unit 121 based on the QP map information shown in FIG. 6A, showing attention map information (attention map sequence) for each frame in a time series of times T = t1, t2, t3, .... In the QP map at each time, F1 to F3 indicate the area of ​​the entire frame. At this time, the hatched areas H1 to H3 were determined by the above-mentioned determination to be areas where the degree of bit rate reduction is equal to or greater than a predetermined threshold, and therefore were excluded from the area in the attention map sequence. In this example, weighting is performed so that the weight of each pixel information in the excluded area in the attention map sequence is "0," and the weight of each pixel information in each pixel in the other areas is "1."

[0054] Note that pixel information refers to values ​​stored for a specific unit area in a frame of an image or attention map. For example, pixel values ​​(such as actual RGB values ​​stored in each pixel of an image) may be used, but are not limited to these. The attention map generation unit 121 uses the QP map sequence to define weighting as described above for each unit area in each frame of the time series, so that the weight is either "0" or "1." For example, the attention map generation unit 121 may set the hatched area H1 as a single unit area and define the weight of that area as "0." Alternatively, the attention map generation unit 121 may set the unit areas so that the hatched area H1 is composed of multiple unit areas, and define the weight of each unit area as "0." In this case, the unit area is composed of one or multiple pixels. The attention map generation unit 121 outputs this attention map information to the feature integration unit 122.

[0055] The feature integration unit 122 corresponds to the integration unit 12 according to the first embodiment and integrates the generated attention map information with the video. The feature integration unit 122 may generate integrated data by, for example, multiplying each pixel information of the attention map information at each time by each pixel information (e.g., pixel value information) of the corresponding video. In the example of attention map information described above, the weight of each pixel information in the excluded region is "0," so the information for each pixel in this region is also "0" in the integrated data. Therefore, the integrated data includes an image in which the excluded region is masked, and this image represents the region that should be focused on for recognition processing.

[0056] The feature integration unit 122 outputs the integrated data in which the attention areas have been weighted in time and space to the behavior recognition unit 114. The behavior recognition unit 114 performs recognition processing based on this integrated data. In this recognition processing, areas other than the attention areas are prevented from being targeted for recognition processing, and areas of high-quality video that are easy to analyze are targeted for recognition processing. This not only increases the accuracy of the recognition processing, but also reduces the processing load of the recognition processing.

[0057] 7 is a flowchart showing an example of a typical process of the center server 104, and this flowchart will be used to explain an overview of the process of the center server 104. Note that the details of each process are as described above, and therefore will not be explained again.

[0058] First, the video acquisition unit 111 acquires video transmitted from each terminal 101 and QP map information corresponding to the video (step S21; acquisition step). The QP map information acquisition unit 112 extracts QP map information from the information acquired from the video acquisition unit 111 (step S22; extraction step).

[0059] The attention map generation unit 121 generates attention map information using the extracted QP map (step S23; generation step). The feature integration unit 122 integrates the generated attention map information with the video to generate integrated data (step S24; integration step). The behavior recognition unit 114 executes recognition processing based on the integrated data (step S25; recognition step).

[0060] [Explanation of Effects] As described above, the attention map generation unit 121 generates attention map information (image quality feature information) indicating spatiotemporal features using QP map information (image quality information) indicating the image quality of the video. The feature integration unit 122 generates integrated data by integrating the video and the attention map information, and the behavior recognition unit 114 performs recognition processing on subjects included in the video based on the integrated data. The behavior recognition unit 114 can perform recognition processing after identifying areas in the video where the bit rate significantly decreases. This makes it possible to suppress the impact of changes in image quality that occur in the video and improve the accuracy of video recognition.

[0061] Furthermore, the attention map generation unit 121 may generate attention map information indicating weights of pixel information in video frames based on the QP map information. The feature integration unit 122 generates integrated data, in which the pixels of the video frames have been weighted based on the attention map information. This allows the behavior recognition unit 114 to analyze the integrated data using a method similar to that used for normal video, eliminating the need for a special behavior recognition function to be installed in the center server 104 and reducing costs.

[0062] Furthermore, QP map information, which indicates the degree of compression of a frame region included in the video, may be used as image quality information indicating the image quality of the video. This prevents the behavior recognition unit 114 from analyzing regions with a high degree of compression. Therefore, as described above, it is possible to increase the accuracy of the recognition process and reduce the processing load of the recognition process.

[0063] The behavior recognition unit 114 may recognize the behavior of the subject. For the reasons described above, the behavior recognition unit 114 can determine the behavior of the subject with high accuracy.

[0064] In (2A), the attention map generation unit 121 can generate attention map information from QP map information by determining an algorithm based on a rule base using a threshold, as described above. However, the attention map generation unit 121 may also be a pre-trained AI model (e.g., a neural network). This training is performed by inputting training data including sample QP map information and correct labels indicating attention map information corresponding to each frame of the sample QP map information into the AI ​​model. This method also allows the attention map generation unit 121 to generate attention map information in which areas where accurate recognition processing is considered difficult are masked.

[0065] Below, (2B) and (2C) describe variations of (2A).

[0066] (2B) In (2A), the attention map generation unit 121 generated attention map information by masking areas where the degree of decrease in bit rate from the reference value is equal to or greater than a predetermined threshold. However, even such areas may be useful for behavior recognition processing in some cases. Therefore, in (2B), a variation for generating attention map information that takes such areas into consideration will be described.

[0067] In detail, in (2A), the attention map generation unit 121 masks an area where the degree of bit rate reduction is equal to or greater than a predetermined threshold by setting the weight of each pixel information of the area to "0." However, the attention map generation unit 121 does not necessarily set the weight of the pixel information of the area where the degree of bit rate reduction is equal to or greater than a predetermined threshold to "0," but may set the weight to a value greater than 0 and less than 1. In this case, the weight of the information of the area where the degree of bit rate reduction is equal to or greater than a predetermined threshold is low, but the area is still subject to recognition processing by the behavior recognition unit 114.

[0068] In this example, the attention map generation unit 121 is a neural network that has been trained in advance. During training of this neural network, a sample video containing a plurality of sample images is input to the center server 104 as the video. The video acquisition unit 111 to the behavior recognition unit 114 of the center server 104 perform the above-described processing on the acquired sample video. At this time, the attention map generation unit 121 is trained so that a loss function calculated based on the recognition result of the behavior recognition unit 114 and the correct answer label of the behavior recognition corresponding to the sample video is equal to or less than a predetermined threshold. For example, the loss function may be trained to have the smallest possible value. The loss function may be, for example, cross-entropy loss or mean squared error, but is not limited to these. Through this training, the weighting settings in the attention map generation unit 121 are updated so that the weight of pixel information is a value other than "0" depending on the situation, even in areas where the degree of bit rate reduction is equal to or greater than a predetermined threshold.

[0069] The feature integration unit 122 integrates the attention map information generated by the attention map generation unit 121 as described above with the video. As described above, the feature integration unit 122 generates integrated data by, for example, multiplying each pixel information of the attention map information at each time by each pixel information of the corresponding video. The integrated data generated by the feature integration unit 122 can be said to be video weighted according to the degree of attention of the attention area in time and space. The behavior recognition unit 114 performs recognition processing on this integrated data.

[0070] In the example described above, even in regions where the degree of bit rate reduction is equal to or greater than a predetermined threshold, the weighting of pixel information can be flexibly set without uniformly targeting the regions for masking. This can further improve the accuracy of the recognition process by the behavior recognition unit 114. Furthermore, even in regions where the degree of bit rate reduction is less than a predetermined threshold as a result of learning, the attention map generation unit 121 can set the weight of the pixel information of the region to a value greater than 0 and less than 1, rather than necessarily setting the weight to "1." This prevents the attention map generation unit 121 from targeting such regions in the recognition process by the behavior recognition unit 114. This can enable the recognition process to be performed efficiently. For example, as a result of learning, the attention map generation unit 121 can set the weight of each pixel information based on information on the degree of spatiotemporal bit rate fluctuation in the QP map sequence.

[0071] In (2B), the attention map generation unit 121 may not be a neural network, but may be another type of AI model that has been trained in advance. Furthermore, the attention map generation unit 121 may set regions where the weight of pixel information is a value other than "0" or "1" by rule-based judgment instead of an AI model. For example, two types of judgment thresholds may be set, and for regions where the degree of decrease in bit rate from the reference value is equal to or greater than a first threshold Th1 and less than a second threshold Th2 (Th2 > Th1), the weight of each pixel information in the region may be set to a value greater than 0 and less than 1. Three or more types of thresholds may also be set. In this way, the attention map generation unit 121 may use any method to gradually determine the weight of pixel information based on the degree of decrease in bit rate from the reference value.

[0072] (2C) In (2A) and (2B), the image is integrated with the attention map information in the feature integration unit 122. However, the feature integration unit 122 may generate integrated data by integrating the attention map information with image feature information that indicates the spatiotemporal features of the image.

[0073] Fig. 8 is a block diagram showing another example of a compressed information integration unit. In the compressed information integration unit 113 shown in Fig. 8, the feature information generation unit 120 further includes a video feature extraction unit 123 in addition to an attention map generation unit 121. Each unit will be described below.

[0074] As shown in (2A), the attention map generation unit 121 generates attention map information (image quality feature information) indicating spatiotemporal features using QP map information indicating the image quality of the video. The attention map generation unit 121 outputs the attention map information to the feature integration unit 122.

[0075] Here, the attention map generating unit 121 may be a neural network that has been trained in advance, as shown in (2B). The training of this neural network is as explained in (2B), so the explanation is omitted here.

[0076] The video feature extraction unit 123 generates video feature information indicating the features of an image for each frame at each time of the video, and outputs the video feature information to the feature integration unit 122. The video feature information can be expressed as, for example, a feature matrix.

[0077] In this example, the video feature extraction unit 123 is a neural network that has been trained in advance. When training this neural network, a sample video containing a plurality of sample videos is input as the video to the center server 104. The video acquisition unit 111 to the behavior recognition unit 114 of the center server 104 perform the above-described processing on the acquired sample video. At this time, the video feature extraction unit 123 is trained so that a loss function calculated based on the recognition result of the behavior recognition unit 114 and the correct answer label of the behavior recognition corresponding to the sample video is equal to or less than a predetermined threshold. For example, the loss function may be trained so as to have the smallest value among the values ​​that the function can take. The loss function may be, for example, cross-entropy loss or mean squared error, but is not limited to these.

[0078] The feature integration unit 122 generates integrated data by integrating the attention map information and the video feature information. The feature integration unit 122 may generate the integrated data by, for example, adding each pixel information of the attention map information at each time and each pixel information of the corresponding video feature information. This emphasizes features in the image in time and space as feature amounts and reflects them in the integrated data. However, the feature integration unit 122 may generate the integrated data by processing other than addition. The feature integration unit 122 outputs the generated integrated data to the behavior recognition unit 114.

[0079] As another example, the feature integration unit 122 may be implemented by a pre-trained AI model rather than by rule-based processing. For example, the feature integration unit 122 may be implemented by a neural network. When this neural network is trained, sample video including multiple sample videos is input to the center server 104 as video. The video acquisition unit 111 to the behavior recognition unit 114 of the center server 104 perform the above-described processing on the acquired sample video. At this time, the feature integration unit 122 is trained so that a loss function calculated based on the recognition result of the behavior recognition unit 114 and the correct answer label of the behavior recognition corresponding to the sample video is equal to or less than a predetermined threshold. For example, the loss function may be trained to have the smallest possible value. The loss function may be, for example, cross-entropy loss or mean squared error, but is not limited thereto.

[0080] With the configuration described above, the behavior recognition unit 114 performs recognition processing on integrated data in which attention map information and video feature information are integrated. At this time, since the video feature information is already indicated in the integrated data, there is no need for the behavior recognition unit 114 to perform processing to extract image features. This allows for simplification of the function of the behavior recognition unit 114.

[0081] Furthermore, the video feature extraction unit 123 that generates the video feature information can be configured with a trained neural network, which enables features in the video to be captured with high accuracy, thereby improving the accuracy of behavior recognition by the behavior recognition unit 114.

[0082] In (2C), the video feature extraction unit 123 may be a different type of AI model that has undergone pre-training instead of a neural network. Also, the video feature extraction unit 123 may generate video feature information indicating the features of an image for each frame by making a rule-based determination.

[0083] The technical concept of the present disclosure is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0084] For example, in the second embodiment, at least one of lightness information and luminance information in the video may be used instead of or in addition to the QP map information. In areas of the video where the brightness is higher than a predetermined threshold, the accuracy of video recognition may be reduced. Therefore, by generating image quality feature information using lightness information or luminance information and performing recognition processing on integrated data reflecting the image quality feature information, it is possible to suppress the influence of high brightness areas on the recognition processing even when the video includes high brightness areas.

[0085] In (2A) and (2B), the weight of each pixel information in the attention map information generated by the attention map generation unit 121 is a value between 0 and 1. However, the value that the weight of each pixel information can take is not limited to this. For example, the weight of each pixel information may be set to a value between 0 and an arbitrary positive number, or may be set to a negative value.

[0086] In the MEC server 103, information on the bit rate assigned to each terminal 101 may be transmitted from the MEC server 103 to the center server 104. The attention map generation unit 121 may change parameters for generating attention map information for video transmitted from each terminal 101 based on the bit rate. For example, as shown in (2A) and (2B), when the attention map generation unit 121 determines whether there is an area where the degree of decrease in bit rate from the reference value is equal to or greater than a predetermined threshold, the attention map generation unit 121 can change at least one of the reference value or the threshold in accordance with the change in bit rate. As an example, when the bit rate assigned to the terminal 101A decreases, the attention map generation unit 121 may lower the reference value and threshold for the above determination regarding the video of the terminal 101A. In this way, the attention map generation unit 121 can make a determination for each terminal 101 taking into account the bit rate of the entire video and generate a highly accurate attention map. Therefore, the behavior recognition unit 114 can perform recognition processing with high accuracy.

[0087] The center server 104 may output alert information based on the recognition result of the behavior recognition unit 114. For example, when the behavior recognition unit 114 determines that a person in the video is performing a predetermined behavior, the center server 104 can present alert information on an interface such as a screen. The center server 104 can also display a GUI (Graphical User Interface) on the screen of its display unit and display the video acquired from the terminal 101, the recognition result of the behavior recognition unit 114, an alert, etc. on the GUI.

[0088] In the second embodiment, the compressed information integration unit 113 and the behavior recognition unit 114 are provided in a single device, the center server 104. However, any part of the processing of the compressed information integration unit 113 and the behavior recognition unit 114 may be executed by another device, not by the center server 104. In other words, as described in (1B) of the first embodiment, the processing of the compressed information integration unit 113 and the behavior recognition unit 114 may be realized as a system distributed among multiple devices.

[0089] In the above-described embodiments, this disclosure has been described as a hardware configuration, but this disclosure is not limited to this. This disclosure can also be realized by having a processor in a computer execute a computer program to perform the processes (steps) of the video processing device, the device in the video processing system, or the center server described in the above-described embodiments.

[0090] 9 is a block diagram showing an example of the hardware configuration of an information processing device 90 that executes the processes of the above-described embodiments. Referring to FIG. 9, the information processing device 90 includes a signal processing circuit 91, a processor 92, and a memory 93.

[0091] The signal processing circuit 91 is a circuit for processing signals in accordance with the control of the processor 92. The signal processing circuit 91 may include a communication circuit for receiving signals from a transmitting device.

[0092] The processor 92 is connected (coupled) to the memory 93, and performs the processing of the device described in the above embodiment by reading and executing software (computer programs) from the memory 93. As an example of the processor 92, one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Demand-Side Platform), and an ASIC (Application Specific Integrated Circuit) may be used, or a plurality of these may be used in parallel.

[0093] The memory 93 may be a volatile memory, a nonvolatile memory, or a combination thereof. The memory 93 is not limited to one, and may be provided in multiple units. The volatile memory may be, for example, a random access memory (RAM) such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The nonvolatile memory may be, for example, a read only memory (ROM) such as a programmable random only memory (PROM) or an erasable programmable read only memory (EPROM), a flash memory, or a solid state drive (SSD).

[0094] The memory 93 is used to store one or more instructions. Here, the one or more instructions are stored as a group of software modules in the memory 93. The processor 92 can perform the processes described in the above embodiments by reading and executing the group of software modules from the memory 93.

[0095] The memory 93 may include a memory provided outside the processor 92, as well as a memory built into the processor 92. The memory 93 may also include a storage device located away from the processors constituting the processor 92. In this case, the processor 92 can access the memory 93 via an I / O (Input / Output) interface.

[0096] As described above, one or more processors included in each device in the above-described embodiments execute one or more programs including instructions for causing a computer to execute the algorithms described using the drawings. This processing enables the information processing described in each embodiment to be realized.

[0097] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disk (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0098] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) An image processing device comprising: a feature information generation unit that generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of an image; an integration unit that generates integrated data that integrates information about the image including the spatiotemporal features of the image and the image quality feature information generated by the feature information generation unit; and a recognition unit that performs recognition processing on an object included in the image based on the integrated data. (Supplementary Note 2) The image processing device according to Supplementary Note 1, wherein the feature information generation unit generates the image quality feature information that indicates weights of pixel information in frames of the image based on the image quality information, and the integration unit generates an image in which pixels of the frames of the image are weighted based on the image quality feature information as the integrated data. (Supplementary Note 3) The video processing device according to Supplementary Note 1, wherein the feature information generation unit generates the image quality feature information indicating a map of feature amounts in time and space of the image quality information, and the integrating unit generates the integrated data by integrating the image quality feature information and video feature information, which is information about the video and indicates features in time and space of the video. (Supplementary Note 4) The video processing device according to Supplementary Note 3, wherein the feature information generation unit further generates the video feature information based on the video. (Supplementary Note 5) The video processing device according to any one of Supplements 1 to 4, wherein, when a sample video serving as a sample is acquired as the video, the feature information generation unit has a neural network trained so that a loss function calculated based on the recognition result of the recognition unit and a correct label of action recognition corresponding to the sample video is equal to or less than a predetermined threshold. (Supplementary Note 6) The video processing device according to any one of Supplements 1 to 5, wherein the image quality information is information indicating a compression degree of a region of a frame included in the video. (Supplementary Note 7) The image processing device according to any one of Supplementary Notes 1 to 6, wherein the recognition unit recognizes a behavior of the subject.(Supplementary Note 8) A video processing system comprising: a feature information generation unit that generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of a video; an integration unit that generates integrated data that integrates information about the video including the spatiotemporal features of the video and the image quality feature information generated by the feature information generation unit; and a recognition unit that performs recognition processing on a subject included in the video based on the integrated data. (Supplementary Note 9) The video processing system according to Supplementary Note 8, wherein the feature information generation unit generates the image quality feature information that indicates weights of pixel information in frames of the video based on the image quality information, and the integration unit generates video in which pixels of the video have been weighted based on the image quality feature information. (Supplementary Note 10) The video processing system according to Supplementary Note 8, wherein the feature information generation unit generates the image quality feature information that indicates a map of feature amounts in spatiotemporal of the image quality information, and the integration unit generates the integrated data that integrates the image quality feature information and video feature information related to the video that indicates the spatiotemporal features of the video. (Supplementary Note 11) The video processing system according to Supplementary Note 10, wherein the feature information generation unit further generates the video feature information based on the video. (Supplementary Note 12) The video processing system according to any one of Supplements 8 to 11, wherein the feature information generation unit has a neural network trained so that, when a sample video serving as a sample is acquired as the video, a loss function calculated based on the recognition result of the recognition unit and a correct label of action recognition corresponding to the sample video is equal to or less than a predetermined threshold. (Supplementary Note 13) The video processing system according to any one of Supplements 8 to 12, wherein the image quality information is information indicating a degree of compression of an area of ​​a frame included in the video. (Supplementary Note 14) The video processing system according to any one of Supplements 8 to 13, wherein the recognition unit recognizes an action of the subject.(Supplementary Note 15) A computer-executed video processing method comprising: generating image quality feature information indicating spatiotemporal features of image quality information indicating the image quality of a video; generating integrated data by integrating the image quality feature information with information about the video including the spatiotemporal features of the video; and performing recognition processing for a subject included in the video based on the integrated data. (Supplementary Note 16) The video processing method of Supplementary Note 15, generating image quality feature information indicating weights of pixel information in frames of the video based on the image quality information, and generating an image in which pixels of the video have been weighted based on the image quality feature information as the integrated data. (Supplementary Note 17) The video processing method of Supplementary Note 15, generating image quality feature information indicating a map of spatiotemporal feature amounts of the image quality information, and generating the integrated data by integrating the image quality feature information with video feature information related to the video, the video feature information indicating the spatiotemporal features of the video. (Supplementary Note 18) The video processing method of Supplementary Note 17, generating the video feature information based on the video. (Supplementary Note 19) The video processing method according to any one of Supplements 15 to 18, wherein, when a sample video serving as a sample is input as the video, learning is performed so that a loss function calculated based on the recognition result of the recognition processing and a correct label of action recognition corresponding to the sample video is equal to or less than a predetermined threshold. (Supplementary Note 20) The video processing method according to any one of Supplements 15 to 19, wherein the image quality information is information indicating a compression degree of an area of ​​a frame included in the video. (Supplementary Note 21) The video processing method according to any one of Supplements 15 to 20, wherein the recognition processing recognizes the action of the subject. (Supplementary Note 22) A non-transitory computer-readable medium having stored thereon a program causing a computer to execute the steps of: generating image quality feature information indicating spatiotemporal features of image quality information indicating image quality of the video; generating integrated data by integrating the image quality feature information with information related to the video including the spatiotemporal features of the video; and performing recognition processing for the subject included in the video based on the integrated data.

[0099] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the disclosure.

[0100] REFERENCE SIGNS LIST 10 Video processing device 11 Feature information generation unit 12 Integration unit 13 Recognition unit 20 Video processing system 21 Feature information generation device 22 Recognition device 100 Video recognition system 101 Terminal 102 Base station 103 MEC server 104 Center server 111 Video acquisition unit 112 QP map information acquisition unit 113 Compression information integration unit 114 Action recognition unit 120 Feature information generation unit 121 Attention map generation unit 122 Feature integration unit 123 Video feature extraction unit

Claims

1. a feature information generating unit that generates image quality feature information that indicates spatiotemporal features of image quality information that indicates an image quality of a video; an integration unit that generates integrated data by integrating information about the image including spatiotemporal features of the image and the image quality feature information generated by the feature information generation unit; a recognition unit that executes a recognition process for a subject included in the video based on the integrated data; A video processing device comprising:

2. the feature information generating unit generates the image quality feature information indicating a weight of pixel information in the frame of the video based on the image quality information; The integration unit generates, as the integrated data, an image in which pixels of the frame of the image are weighted based on the image quality characteristic information. The video processing device according to claim 1 .

3. the feature information generating unit generates the image quality feature information indicating a map of a feature amount in time and space of the image quality information; the integration unit generates the integrated data by integrating the image quality characteristic information and image characteristic information, which is information related to the image and indicates spatiotemporal characteristics of the image. The video processing device according to claim 1 .

4. The feature information generating unit further generates the video feature information based on the video. The image processing device according to claim 3 .

5. a feature information generating unit that generates image quality feature information that indicates spatiotemporal features of image quality information that indicates an image quality of a video; an integration unit that generates integrated data by integrating information about the image including spatiotemporal features of the image and the image quality feature information generated by the feature information generation unit; a recognition unit that executes a recognition process for a subject included in the video based on the integrated data; A video processing system comprising:

6. the feature information generating unit generates the image quality feature information indicating a weight of pixel information in the frame of the video based on the image quality information; The integration unit generates, as the integrated data, an image in which pixels of the frame of the image are weighted based on the image quality characteristic information. The video processing system according to claim 5 .

7. the feature information generating unit generates the image quality feature information indicating a map of a feature amount in time and space of the image quality information; the integration unit generates the integrated data by integrating the image quality characteristic information and image characteristic information, which is information related to the image and indicates spatiotemporal characteristics of the image. The video processing system according to claim 5 .

8. The feature information generating unit further generates the video feature information based on the video. The video processing system according to claim 7.

9. generating image quality characteristic information indicating spatiotemporal characteristics of image quality information indicating image quality of the video; generating integrated data by integrating information about the image including spatiotemporal characteristics of the image and the image quality characteristic information; and executing a recognition process for a subject included in the video based on the integrated data. A computer-implemented image processing method.

10. A method for generating image quality characteristic information indicating spatiotemporal characteristics of image quality information indicating image quality of a video, generating integrated data by integrating information about the image including spatiotemporal characteristics of the image and the image quality characteristic information; and executing a recognition process for a subject included in the video based on the integrated data. A program that causes a computer to do something.

Images