An image processing method integrating time information

Abstract

The present invention discloses an image processing method incorporating time information, and belongs to the field of communication technology. The present invention utilizes captured video data to extract image data of each frame or a specific frame of the video data, and then crops all images into a uniform size to satisfy the input of a neural network. In another step, all images are grayed and converted into single-channel images. In the next step, each three adjacent frames of single-channel images are deeply superimposed to synthesize a three-channel non-RGB image, and this step is repeated to synthesize all single-channel images. Finally, all the synthesized images are formed into a new image data set and annotated. This new data set is applied to a neural network and achieves good prediction results.

Description

Technical Field

[0001] The invention belongs to the field of data processing, and in particular relates to an image processing method integrating time information. Background Art

[0002] In the field of image data processing, images used for data analysis only contain data from a certain scene at a certain moment. A large data set is often just a collection of images at a large number of discrete time points. For example, neural networks have been widely used in the field of artificial intelligence. When using these traditional data to train neural network models, general object classification tasks and instance segmentation tasks may produce relatively good results. However, in the field of human action recognition, the experimental results obtained from such traditional data sets are not ideal, or even very poor.

[0003] For example, in the field of behavioral posture recognition, there is a problem that cannot be solved by traditional images that only contain information at a single time point: the image does not contain time information, and the judgment of behavioral posture only has spatial basis, not temporal basis.

[0004] For example, in an image of a person holding a door handle, it is impossible to determine whether the person is opening or closing the door based on spatial evidence alone.

[0005] Traditional data processing methods that only have spatial information are no longer applicable to the recognition of behavioral postures, and are very likely to cause misjudgment of human behavioral postures. In addition, the training results of the data sets obtained by traditional data processing methods in the neural network model are also not ideal, and the accuracy rate is often low and the loss value is high. Summary of the invention

[0006] In view of the above deficiencies in the prior art, the technical problem to be solved by the present invention is to propose an image processing method that incorporates time information. Through this new image processing method, an image that originally only contains the scene features of a certain moment is changed into an image under a certain scene feature that contains time information. The time information contained in each channel of the image effectively improves the accuracy of human behavior posture recognition based on neural networks, and while improving the accuracy, the accuracy of shopping recommendations for customers is also improved, thereby better meeting customer needs.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: an image processing method incorporating time information, comprising the following steps:

[0008] Step 1: Collect motion status video data through video acquisition equipment.

[0009] Step 2: Extract the images of the collected video data frame by frame or at a certain time interval t and save the extracted appropriate video frames as the original image data set.

[0010] Step 3: Crop all images of the original dataset to meet the requirements of the later stage.

[0011] Step 4: All images in the current dataset are three-channel RGB images. All images are grayscaled to obtain single-channel images of appropriate size.

[0012] Step 5: Superimpose the depth of every three adjacent frames or several frames of single-channel images with a certain time interval t to generate a three-channel or multi-channel non-RGB image.

[0013] Step 6: Repeat the process of step 5 to synthesize a large number of three-channel or multi-channel non-RGB images to form an image dataset that incorporates time information. Since this example is applied to the shopping recommendation scenario, the dataset is named shopping recommendation dataset.

[0014] Step 7: Annotate and label all synthetic images of the shopping recommendation dataset.

[0015] In the above method, in the process of extracting the images frame by frame from the video data in step 2, each frame of the image is sorted by time according to the extraction time, so as to ensure the continuity of the customer's actions in the images of the three channels in the synthesized image.

[0016] The three-channel non-RGB image synthesized in step 5 contains the temporal information of customer actions in the shopping scene, and can preserve the customer's behavioral characteristics more fully and accurately.

[0017] In step seven, the synthesized image is labeled using onehot encoding.

[0018] The beneficial effects of the present invention are as follows: an image processing method incorporating time information provided by the present invention is different from a traditional three-channel RGB image that only contains information of a single moment. The new image processing method incorporating time information first unifies the sizes of traditional images that are consecutive or spaced a certain distance apart, and then converts them into single-channel grayscale images respectively. Next, the grayscale images of each three adjacent time points are superimposed in depth to synthesize a new three-channel non-RGB image. After that, new images are synthesized in sequence according to the time sequence of the single-channel grayscale images, and a large number of synthesized new images are formed into a new data set, which is named the shopping recommendation data set in the shopping recommendation scenario of this example, and each newly synthesized image is labeled. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The following is a brief description of the contents and symbols in the drawings of this specification:

[0020] Figure 1 It is a flowchart of an evaluation method according to a specific embodiment of the present invention.

[0021] Figure 2 It is a flowchart of the new image processing method.

[0022] Figure 3 is a sample example diagram of the synthetic data set of the present invention.

[0023] Figure 4 This is a graph of the shopping recommendation prediction accuracy obtained by training a new synthetic image dataset on a general convolutional neural network as a function of the number of training times.

[0024] Figure 5 This is a graph of the shopping recommendation prediction loss value obtained by training a new synthetic image dataset on a general convolutional neural network as a function of the number of training times. DETAILED DESCRIPTION

[0025] The present invention starts from a new image data processing method, generates an image data set containing time information, applies it to a shopping recommendation algorithm based on a convolutional neural network, and achieves good prediction results.

[0026] like Figure 1 FIG. 1 is a flow chart of an image processing method incorporating time information according to the present invention, comprising the following steps:

[0027] Step 1: Collect motion status video data through video acquisition equipment.

[0028] Step 2: Extract the images of the collected video data frame by frame or at a certain time interval t and save the extracted appropriate video frames as the original image dataset Dataset.

[0029] Step 3: Crop all images of the original dataset to meet the requirements of the later stage.

[0030] Step 4: All images in the current dataset are three-channel RGB images. All images are grayscaled to obtain single-channel images of appropriate size.

[0031] Step 5: Superimpose every three adjacent frames or several single-channel images at a certain time interval t in depth to generate a three-channel or multi-channel non-RGB image newimg1.

[0032] Step 6: Repeat the process of step 5 to synthesize a large number of three-channel or multi-channel non-RGB images to form an image dataset that incorporates time information. Since this example is applied to the shopping recommendation scenario, the dataset is named shopping recommendation dataset.

[0033] Step 7: Annotate and label all synthetic images of the shopping recommendation dataset.

[0034] Figure 2 The synthesis process of three consecutive frames of images into a three-channel non-RGB image is demonstrated.

[0035] Figure 3 These are some sample images of a synthetic image data set obtained by the new image data processing method of the present invention.

[0036] Figure 4 The study shows how the shopping recommendation prediction accuracy of a new synthetic image dataset trained on a general convolutional neural network changes with the number of training times, with a peak accuracy of 97% and a stable accuracy of 96%.

[0037] Figure 5 The paper shows how the shopping recommendation prediction loss value obtained by training a new synthetic image dataset on a general convolutional neural network changes with the number of training times. The loss value eventually drops to an extremely low level of only 0.1 to 0.2.

[0038] The present invention utilizes an image processing method that incorporates time information to greatly improve the accuracy of predicting human behavior and posture, allowing the algorithm to accurately provide customers with the goods they want to buy in a shopping scenario, which has great application value.

[0039] The present invention is described above by way of example in conjunction with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-mentioned manner. As long as various non-substantial improvements are made by using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope defined in the claims.

Claims

1. An image processing method incorporating time information, characterized in that: The following steps are involved: Step 1: Collect the motion state data of the video object through the video acquisition device; Step 2: extract the collected video data frame by frame or at a certain time interval t and save the extracted appropriate video frames as the original image data set; Step 3: Crop all images of the original dataset to meet the requirements of the later stage; Step 4: All images in the current dataset are three-channel RGB images. All images are grayscaled to obtain single-channel images of appropriate size. Step 5: Superimpose every three adjacent frames or several single-channel images at a certain time interval t in depth to generate a three-channel or multi-channel non-RGB image newimg1; Step 6: Repeat the process of step 5 to synthesize a large number of three-channel or multi-channel non-RGB images to form an image dataset that incorporates time information. Since this example is applied to the shopping recommendation scenario, the dataset is named shopping recommendation dataset; Step 7: Annotate and label all synthetic images of the recommended dataset.

2. The image processing method incorporating time information according to claim 1, characterized in that: In step 2, during the image extraction process of the video data frame by frame or at a certain time interval t, each extracted frame image is sorted by time according to the order of extraction time to ensure the continuity of the object motion state in the images of the three channels in the synthesized image.

3. The image processing method incorporating time information according to claim 1, characterized in that: In step 5, the original images img1, img2, img3 in the previous step are grayed and converted to Gimg1, Gimg2, and Gimg3, which are placed in the three channels p1, p2, and p3 of the new image respectively. The corresponding relationship of the generated images is The synthesized three-channel non-RGB image newimg1 is It contains the time information of the motion state of an object in a scene, which can more fully and accurately preserve the motion characteristics of the object in the scene; the calculation method of converting the grayscale image of a 3*3 image into the pixel value of a three-channel non-RGB image is as follows: Note: If are the pixel values ​​on the three channels.

4. The image processing method incorporating time information according to claim 1, characterized in that: In step seven, the synthesized image is labeled using onehot encoding.

Images