Image data processing method and device, equipment and storage medium

Abstract

The invention discloses an image data processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring a first original image of a sample object, and determining a second original image based on the first original image; the first original image and the second original image are three-channel images; amplifying the first original image based on the image of each channel corresponding to the first original image to obtain a plurality of new first images, and amplifying the second original image based on the image of each channel corresponding to the second original image to obtain a plurality of new second images; constructing an original sample pair based on the first original image and the second original image, and constructing a plurality of amplified sample pairs based on the plurality of new first images and the plurality of new second images; the original sample pair and the plurality of amplified sample pairs are jointly used for training the twin network to be trained. In this way, the generalization ability of the trained twin network can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to an image data processing method, device, equipment and storage medium. Background Art

[0002] At present, the twin network used to identify the behavior of the target object in the vehicle cabin is trained in the following way: face detection is performed on the main driver's seat image frame sequence. If a face is detected, the face key point regression is performed in the face region of interest (ROI); the image of the ROI area is intercepted according to the key points; then, the intercepted image of the ROI area is input into the twin network to be trained, and the twin network to be trained is trained to obtain the twin network used to identify the behavior of the target object in the vehicle cabin. In this way, due to the small amount of data input into the twin network to be trained, the twin network learns fewer features, which in turn leads to poor generalization ability of the trained twin network.

[0003] Therefore, how to improve the generalization ability of the trained twin network has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide an image data processing method, apparatus, device and storage medium, which can improve the generalization ability of a trained twin network.

[0005] In a first aspect, an embodiment of the present application provides an image data processing method, the method comprising:

[0006] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0007] Based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images; and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images;

[0008] An original sample pair is constructed based on the first original image and the second original image, and multiple augmented sample pairs are constructed based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0009] In one embodiment, based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain multiple new first images, and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain multiple new second images, including: performing image division processing on the images of the three channels corresponding to the first original image to obtain multiple first sub-images corresponding to the image of each channel, and performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; for the multiple first sub-images corresponding to the image of each channel of the first original image, a preset number of pixels are selected from each first sub-image to obtain multiple groups of pixels corresponding to the multiple first sub-images, and a new first image is determined based on the multiple groups of pixels; based on the images of the three channels of the first original image, the new first images are amplified to obtain multiple new first images; for the multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image is determined based on the multiple groups of pixels; based on the images of the three channels of the second original image, the new first images are amplified to obtain multiple new first images.

[0010] In one embodiment, image division processing is performed on images of three channels corresponding to the first original image respectively to obtain multiple first sub-images corresponding to the image of each channel, and image division processing is performed on images of three channels corresponding to the second original image respectively to obtain multiple second sub-images corresponding to the image of each channel, including: equally dividing the pixel points of the images of three channels corresponding to the first original image respectively to obtain multiple equally divided first sub-images corresponding to the image of each channel; and equally dividing the pixel points of the images of three channels corresponding to the second original image respectively to obtain multiple equally divided second sub-images corresponding to the image of each channel.

[0011] In one of the embodiments, for multiple first sub-images corresponding to the image of each channel of the first original image, a preset number of pixels are selected from each first sub-image to obtain multiple groups of pixels corresponding to the multiple first sub-images, and a new first image is determined based on the multiple groups of pixels, including: for multiple first sub-images corresponding to the image of each channel of the first original image, a preset number of pixels are selected from each first sub-image to obtain at least one group of pixels corresponding to the multiple first sub-images, and at least one new first image of this round of amplification is determined based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each first sub-image have not been traversed, the steps of selecting a preset number of pixels from each first sub-image to obtain at least one group of pixels corresponding to the multiple first sub-images, and determining at least one new first image of a new round of amplification based on the at least one group of pixels are performed; until all combinations of the preset number of pixels in each first sub-image are traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new first images are obtained.

[0012] In one of the embodiments, for multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image is determined based on the multiple groups of pixels, including: for multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain at least one group of pixels corresponding to the multiple second sub-images, and at least one new second image of this round of amplification is determined based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each second sub-image have not been traversed, the steps of selecting a preset number of pixels from each second sub-image to obtain at least one group of pixels corresponding to the multiple second sub-images, and determining at least one new second image of a new round of amplification based on the at least one group of pixels are performed, until all combinations of the preset number of pixels in each first sub-image are traversed, and a new first image obtained by multiple rounds of amplification is obtained, and multiple new second images are obtained.

[0013] In one embodiment, based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain multiple new first images, and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain multiple new second images, including: for the first original image, updating the pixel value of a preset pixel point in the first channel image to the pixel value of a preset pixel point in the second channel image or the third channel image; updating the pixel value of a preset pixel point in the second channel image to the pixel value of a preset pixel point in the first channel image or the third channel image; updating the pixel value of a preset pixel point in the third channel image to the pixel value of a preset pixel point in the first channel image or the second channel image; based on the first channel image after updating the pixel value, the second channel image after updating the pixel value, and the The third channel image is obtained to obtain a new first image obtained in this round of amplification; a plurality of new first images are obtained from the new first image obtained by multiple rounds of amplification; and, for the second original image, the pixel value of the preset pixel point in the first channel image is updated to the pixel value of the preset pixel point in the second channel image or the third channel image; the pixel value of the preset pixel point in the second channel image is updated to the pixel value of the preset pixel point in the first channel image or the third channel image; the pixel value of the preset pixel point in the third channel image is updated to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after the pixel value is updated, the second channel image after the pixel value is updated, and the third channel image after the pixel value is updated, a new second image obtained in this round of amplification is obtained; a plurality of new second images are obtained from the new second image obtained by multiple rounds of amplification.

[0014] In one embodiment, the method further includes: determining a set of sample image pairs based on the original sample pairs and the multiple amplified sample pairs; for each sample image pair in the set of sample image pairs, inputting the first sample image in the sample image pair into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and inputting the second sample image in the sample image pair into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; determining a total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value, and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image feature and the second The sample image features determine the loss value; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image features into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image features into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the training stop condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the training stop condition to obtain the trained twin network.

[0015] In a second aspect, an embodiment of the present application provides an image data processing device, the device comprising:

[0016] An acquisition and determination module is used to acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0017] An image amplification module, configured to amplify the first original image based on the image of each channel corresponding to the first original image to obtain a plurality of new first images, and to amplify the second original image based on the image of each channel corresponding to the second original image to obtain a plurality of new second images;

[0018] A construction module is used to construct an original sample pair based on a first original image and a second original image, and to construct multiple augmented sample pairs based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train a twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0019] In a third aspect, an embodiment of the present application provides a computing device, including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, the following steps are implemented:

[0020] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0021] Based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images; and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images;

[0022] An original sample pair is constructed based on the first original image and the second original image, and multiple augmented sample pairs are constructed based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0024] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0025] Based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images; and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images;

[0026] An original sample pair is constructed based on the first original image and the second original image, and multiple augmented sample pairs are constructed based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0027] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0028] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0029] Based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images; and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images;

[0030] An original sample pair is constructed based on the first original image and the second original image, and multiple augmented sample pairs are constructed based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0031] The above-mentioned image data processing method, device, vehicle and storage medium, the computing device can obtain the first original image of the sample object, and determine the second original image based on the first original image; the second original image is a local image corresponding to the target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images; based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain multiple new first images, and, based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain multiple new second images; an original sample pair is constructed based on the first original image and the second original image, and multiple amplified sample pairs are constructed based on the multiple new first images and the multiple new second images; the original sample pair and the multiple amplified sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin. By adopting this method, the computing device can obtain a new first image and a new second image by respectively amplifying the first original image and the second original image, and construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image. In this way, when the twin network to be trained is trained using the original sample pair and the amplified sample pair, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a schematic diagram of an application scenario of an image data processing method provided in an embodiment of the present application;

[0034] Figure 2 It is a flowchart of an image data processing method provided in an embodiment of the present application;

[0035] Figure 3 is a flowchart of another image data processing method provided in an embodiment of the present application;

[0036] Figure 4a is a schematic diagram of image division provided in an embodiment of the present application;

[0037] Figure 4b is a schematic diagram of a process for determining a new first image provided by an embodiment of the present application;

[0038] Figure 4c is a schematic diagram of another process of determining a new first image provided by an embodiment of the present application;

[0039] Figure 4d is another schematic diagram of a process for determining a new first image provided by an embodiment of the present application;

[0040] Figure 5 It is a flowchart of another image data processing method provided in an embodiment of the present application;

[0041] Figure 6 is another schematic diagram of a process for determining a new first image provided by an embodiment of the present application;

[0042] Figure 7 It is a structural diagram of a twin network provided in an embodiment of the present application;

[0043] Figure 8 is a structural schematic diagram of an image data processing device provided in an embodiment of the present application;

[0044] Fig. 9 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] First, combine Figure 1 The application scenarios of the image data processing method provided in the embodiment of the present application are introduced. Figure 1 , Figure 1 Schematic diagram of an application scenario of an image data processing method provided in an embodiment of the present application. Figure 1 As shown, the application scenario includes a computing device 101 and a database server 102 .

[0047] The computing device 101 may first obtain a first original image of the sample object from the database server 102, and then the computing device 101 may determine the second original image based on the first original image, that is, obtain the local image of the sample object from the global image of the sample object; further, the computing device 101 may amplify the first original image based on the image of each channel corresponding to the first original image to obtain multiple new first images, and amplify the second original image based on the image of each channel corresponding to the second original image to obtain multiple new second images; finally, the computing device 101 may construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image; wherein the original sample pair and the amplified sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin. By adopting this method, the computing device can obtain a new first image and a new second image by respectively amplifying the first original image and the second original image, and construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image. In this way, when the twin network to be trained is trained using the original sample pair and the amplified sample pair, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training.

[0048] Optionally, the computing device 101 may be a terminal device or a server. The terminal devices mentioned here may include but are not limited to: smart phones, tablet computers, laptop computers, desktop computers, smart watches, smart TVs, smart car terminals, etc. The server mentioned here may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers.

[0049] The following is a detailed description of the image data processing method provided in the embodiment of the present application.

[0050] See also Figure 2 , Figure 2 is a flow chart of an image data processing method provided by an embodiment of the present application. The method can be performed by a vehicle (e.g. Figure 1 The computing device 101 in the embodiment is executed. Figure 2 As shown, the image data processing method may include but is not limited to the following steps:

[0051] S201, obtaining a first original image of a sample object, and determining a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to a behavior to be identified of the sample object; the first original image and the second original image are both three-channel images.

[0052] Optionally, the first original image and the second original image may both be red, green, and blue (RGB) images.

[0053] The sample object may be a driver in a vehicle cabin, and correspondingly, the first original image of the sample object is the first original image of the driver. Assuming that the behavior to be identified is whether the driver smokes, the target area is the area related to the smoking behavior, that is, the driver's mouth area. In this case, the second original image is a partial image corresponding to the mouth area in the first original image of the driver.

[0054] In the case where the sample object is an object in the vehicle cabin, before step S201, the computing device may also control the image acquisition device in the cabin to acquire an image of the sample object, and then process the acquired image to obtain a first original image of the sample object, and store the first original image in the database server. The image acquisition device is, for example, an infrared sensor. The image of the sample object may be a single-channel infrared image of the sample object in the cabin acquired by the infrared sensor. In this case, the computing device may convert the single-channel infrared image into a three-channel infrared image, and use the three-channel infrared image as the first original image of the sample object.

[0055] S202, based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images, and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images.

[0056] When the first original image is an RGB image, the image of each channel corresponding to the first original image includes an R channel image, a G channel image, and a B channel image. Similarly, when the second original image is an RGB image, the image of each channel corresponding to the second original image includes an R channel image, a G channel image, and a B channel image.

[0057] S203, constructing an original sample pair based on the first original image and the second original image, and constructing multiple augmented sample pairs based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0058] In an embodiment of the present application, a computing device may obtain a first original image of a sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; both the first original image and the second original image are three-channel images; based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain multiple new first images, and, based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain multiple new second images; an original sample pair is constructed based on the first original image and the second original image, and multiple amplified sample pairs are constructed based on the multiple new first images and the multiple new second images; the original sample pair and the multiple amplified sample pairs are used together to train a twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin. By adopting this method, the computing device can obtain a new first image and a new second image by respectively amplifying the first original image and the second original image, and construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image. In this way, when the twin network to be trained is trained using the original sample pair and the amplified sample pair, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training.

[0059] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another image data processing method provided in an embodiment of the present application. Figure 2 The image processing method shown differs in that Figure 3The method shown specifically describes how the computing device amplifies the first original image based on the image of each channel corresponding to the first original image to obtain multiple new first images, and amplifies the second original image based on the image of each channel corresponding to the second original image to obtain multiple new second images. Figure 3 As shown, the image data processing method may include but is not limited to the following steps:

[0060] S301, obtaining a first original image of a sample object, and determining a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to a behavior to be identified of the sample object; the first original image and the second original image are both three-channel images.

[0061] In an optional embodiment, the computing device determines the second original image based on the first original image, which may include: inputting the first original image into a face detection model, and when it is detected that the first original image contains a sample object, outputting face information corresponding to the sample object; based on the face information, intercepting a face image corresponding to the sample object from the first original image; inputting the face image into a face key point detection model to obtain the face key point coordinates of the sample object; determining a target area based on the face key point coordinates, and obtaining a second original image corresponding to the target area from the face image.

[0062] In this implementation, the face detection model can be a deep learning model or a machine learning model, which is not limited here.

[0063] In this implementation, the face information may include but is not limited to face coordinates, face height, and face width. The computing device intercepts the face image corresponding to the target object from the first image based on the face information, and may intercept the face image corresponding to the target object from the first image based on the face coordinates, face height, and face width.

[0064] In an optional implementation, other related descriptions of step S301 can refer to the description of the aforementioned step S201, which will not be repeated here.

[0065] S302, performing image division processing on the images of the three channels corresponding to the first original image respectively to obtain multiple first sub-images corresponding to the image of each channel, and performing image division processing on the images of the three channels corresponding to the second original image respectively to obtain multiple second sub-images corresponding to the image of each channel.

[0066] In an optional embodiment, the computing device performs image division processing on the images of the three channels corresponding to the first original image respectively to obtain a plurality of first sub-images corresponding to the image of each channel, and performs image division processing on the images of the three channels corresponding to the second original image respectively to obtain a plurality of second sub-images corresponding to the image of each channel, which may include: equally dividing the pixel points of the images of the three channels corresponding to the first original image respectively to obtain a plurality of equally divided first sub-images corresponding to the image of each channel; and equally dividing the pixel points of the images of the three channels corresponding to the second original image respectively to obtain a plurality of equally divided second sub-images corresponding to the image of each channel.

[0067] For example, assuming that the first original image is a 6*6 RGB image, see Figure 4a , Figure 4a is a schematic diagram of image division provided by an embodiment of the present application. Figure 4a As shown, the computing device may divide the R channel image into four 3*3 first sub-images based on the pixel points of the R channel image of the first original image, that is, Figure 4a The first sub-image 1, the first sub-image 2, the first sub-image 3 and the first sub-image 4 in the first original image. The pixel points in the first sub-image 1 are 0 to 8, the pixel points in the first sub-image 2 are 10 to 18, the pixel points in the first sub-image 3 are 20 to 28, and the pixel points in the first sub-image 4 are 30 to 38. The computing device also divides the G channel image into four 3*3 first sub-images based on the pixel points of the G channel image of the first original image, and divides the B channel image into four 3*3 first sub-images based on the pixel points of the B channel image of the first original image.

[0068] For another example, assuming that the first original image is an 8*8 RGB image, the computing device may divide the first original image into four 4*4 first sub-images by dividing the R channel image into four equal pixels based on the pixels of the R channel image in the first original image; divide the first original image into four 4*4 first sub-images by dividing the G channel image into four equal pixels based on the pixels of the G channel image; and divide the first original image into four 4*4 first sub-images by dividing the B channel image into four equal pixels based on the pixels of the B channel image.

[0069] Among them, the specific process of the computing device equally dividing the pixel points of the images of the three channels corresponding to the second original image to obtain multiple equally divided second sub-images corresponding to the image of each channel is the same as the process of equally dividing the pixel points of the images of the three channels corresponding to the first original image to obtain multiple equally divided first sub-images corresponding to the image of each channel, and will not be repeated here.

[0070] S303, for multiple first sub-images corresponding to the image of each channel of the first original image, select a preset number of pixel points from each first sub-image to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; based on the new first images obtained by respectively amplifying the images of the three channels of the first original image, obtain multiple new first images.

[0071] In an optional embodiment, the computing device selects a preset number of pixels from each first sub-image for each channel image of the first original image, obtains multiple groups of pixels corresponding to the multiple first sub-images, and determines a new first image based on the multiple groups of pixels. This may include: selecting a preset number of pixels from each first sub-image for each channel image of the first original image, obtains at least one group of pixels corresponding to the multiple first sub-images, and determines at least one new first image of this round of amplification based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each first sub-image have not been traversed, executing the steps of selecting a preset number of pixels from each first sub-image, obtains at least one group of pixels corresponding to the multiple first sub-images, and determines at least one new first image of a new round of amplification based on the at least one group of pixels; until all combinations of the preset number of pixels in each first sub-image have been traversed, and a new first image obtained by multiple rounds of amplification is obtained, a plurality of new first images are obtained.

[0072] In this implementation, the preset number may be at least one of the following: 1, 2, 3, . . .

[0073] For example, assuming that the multiple first sub-images corresponding to each channel image of the first original image (such as the R channel image, the G channel image, and the B channel image) are 3*3 images, the preset number can be at least one of the following: 1, 2, 3. That is, the computing device can select 1 pixel from each 3*3 first sub-image for each 3*3 first sub-image corresponding to the R channel image, obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or select 2 pixels from each 3*3 first sub-image, obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or select 2 pixels from each 3*3 first sub-image, obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determine a new first image based on the multiple groups of pixel points. Optionally, for each 3*3 first sub-image corresponding to the G channel image and the B channel image, the computing device determines the new first image in the same manner as the manner of determining the new first image for the 3*3 first sub-image corresponding to the R channel image, which will not be described in detail here.

[0074] When the preset quantity is 1, please refer to Figure 4b , Figure 4b is a schematic diagram of a process for determining a new first image provided by an embodiment of the present application. Figure 4b As shown, the computing device may select one pixel from each first sub-image corresponding to the four R channel images, obtain at least one group of pixels corresponding to the four first sub-images (each group of pixels has four pixels), and determine a new first image based on at least one group of pixels. For example, assuming that the computing device selects 0 pixels from the first sub-image 1, 10 pixels from the first sub-image 2, 20 pixels from the first sub-image 3, and 30 pixels from the first sub-image 4, the computing device may determine based on the four selected pixels. Figure 4b For another example, assuming that the computing device selects pixel 1 from the first sub-image 1, pixel 11 from the first sub-image 2, pixel 21 from the first sub-image 3, and pixel 31 from the first sub-image 4, the computing device may determine based on the selected 4 pixels: Figure 4b For another example, assuming that the computing device selects 2 pixels from the first sub-image 1, 12 pixels from the first sub-image 2, 22 pixels from the first sub-image 3, and 32 pixels from the first sub-image 4, the computing device may determine based on the selected 4 pixels: Figure 4b Image 3 in . Similarly, the computing device can also select 1 pixel point from each first sub-image corresponding to the 4 G channel images, obtain at least one group of pixel points corresponding to the 4 first sub-images, and determine a new first image based on at least one group of pixel points. Select 1 pixel point from each first sub-image corresponding to the 4 B channel images, obtain at least one group of pixel points corresponding to the 4 first sub-images, and determine a new first image based on at least one group of pixel points. It can be understood that when the preset number is 1, the number of new first images corresponding to each channel image after amplification that the computing device can obtain is

[0075] When the preset quantity is 2, see Figure 4c , Figure 4c FIG. 1 is another schematic diagram of a process for determining a new first image provided by an embodiment of the present application. Figure 4cAs shown, the computing device can select 2 pixels from each of the 4 first sub-images corresponding to the R channel image, obtain at least one group of pixels corresponding to the 4 first sub-images (each group of pixels has 8 pixels), and determine a new first image based on at least one group of pixels. For example, the computing device can select 2 pixels from each of the 4 first sub-images in the horizontal direction, such as 0 and 1 pixels selected from the first sub-image 1; 10 and 11 pixels selected from the first sub-image 2; 20 and 21 pixels selected from the first sub-image 3; and 30 and 31 pixels selected from the first sub-image 4. In this case, the computing device can determine the new first image based on the 4 selected pixels. Figure 4c For another example, the computing device may select two pixels in the vertical direction from each of the four first sub-images, such as 0 and 3 pixels from the first sub-image 1; 10 and 13 pixels from the first sub-image 2; 20 and 23 pixels from the first sub-image 3; and 30 and 33 pixels from the first sub-image 4. In this case, the computing device may determine based on the four selected pixels: Figure 4c For another example, the computing device may select two pixels in the diagonal and anti-diagonal directions from the four first sub-images, such as 0 and 4 pixels selected from the first sub-image 1; 10 and 14 pixels selected from the first sub-image 2; 20 and 24 pixels selected from the first sub-image 3; and 30 and 34 pixels selected from the first sub-image 4. In this case, the computing device may determine based on the four selected pixels: Figure 4c Image 3 in . Similarly, the computing device can also select 2 pixels from each first sub-image corresponding to the 4 G channel images, obtain at least one group of pixels corresponding to the 4 first sub-images, and determine a new first image based on at least one group of pixels. Select 2 pixels from each first sub-image corresponding to the 4 B channel images, obtain at least one group of pixels corresponding to the 4 first sub-images, and determine a new first image based on at least one group of pixels. It can be understood that when the preset number is 2, the number of new first images corresponding to each channel image after amplification that the computing device can obtain is

[0076] In the case of a preset quantity of 3, see Figure 4d , Figure 4d FIG. 1 is another schematic diagram of a process for determining a new first image provided by an embodiment of the present application. Figure 4dAs shown, the computing device can select 3 pixels from each of the 4 first sub-images to obtain at least one group of pixels corresponding to the 4 first sub-images (each group of pixels has 8 pixels), and determine a new first image based on at least one group of pixels. For example, the computing device can select 3 pixels in the horizontal direction from each of the 4 first sub-images, such as 0, 1, and 2 pixels from the first sub-image 1; 10, 11, and 12 pixels from the first sub-image 2; 20, 21, and 22 pixels from the first sub-image 3; and 30, 31, and 32 pixels from the first sub-image 4. In this case, the computing device can determine the new first image based on the 4 selected pixels. Figure 4d For another example, the computing device may select two pixels in the vertical direction from each of the four first sub-images, such as 0, 3, and 6 pixels from the first sub-image 1; 10, 13, and 16 pixels from the first sub-image 2; 20, 23, and 26 pixels from the first sub-image 3; and 30, 33, and 36 pixels from the first sub-image 4. In this case, the computing device may determine based on the four selected pixels: Figure 4d For another example, the computing device may select two pixels in the diagonal and anti-diagonal directions from the four first sub-images, such as 0, 4, and 8 pixels selected from the first sub-image 1; 10, 14, and 18 pixels selected from the first sub-image 2; 20, 24, and 28 pixels selected from the first sub-image 3; and 30, 34, and 38 pixels selected from the first sub-image 4. In this case, the computing device may determine based on the four selected pixels: Figure 4d Image 3 in . Similarly, the computing device can also select 3 pixels from each first sub-image corresponding to the 4 G channel images, obtain at least one group of pixels corresponding to the 4 first sub-images, and determine a new first image based on at least one group of pixels. Select 3 pixels from each first sub-image corresponding to the 4 B channel images, obtain at least one group of pixels corresponding to the 4 first sub-images, and determine a new first image based on at least one group of pixels. It can be understood that when the preset number is 2, the number of new first images corresponding to each channel image after amplification that the computing device can obtain is

[0077] For another example, assuming that the first sub-image corresponding to each channel image of the first original image (such as the R channel image, the G channel image, and the B channel image) is a 4*4 image, the preset number may be at least one of the following: 1, 2, 3, 4. That is, for each 3*3 first sub-image corresponding to the R channel image, the computing device may select 1 pixel from each 4*4 first sub-image respectively to obtain multiple groups of pixels corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixels; and / or, select 2 pixels from each 4*4 first sub-image respectively to obtain multiple groups of pixels corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixels; and / or, select 2 pixels from each 4*4 first sub-image respectively to obtain multiple groups of pixels corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixels; and / or, select 4 pixels from each 4*4 first sub-image respectively to obtain multiple groups of pixels corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixels. Optionally, for each 4*4 first sub-image corresponding to the G channel image and the B channel image respectively, the computing device determines a new first image in the same way as the way of determining a new first image for the 4*4 first sub-image corresponding to the R channel image, which is not repeated here.

[0078] S304. For the multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image is determined based on the multiple groups of pixels; based on the images of the three channels of the second original image, new first images are obtained by amplifying the images of the three channels respectively to obtain multiple new first images.

[0079] In an optional embodiment, the computing device selects a preset number of pixels from each of the multiple second sub-images corresponding to each channel of the image of the second original image, obtains multiple groups of pixels corresponding to the multiple second sub-images, and determines a new second image based on the multiple groups of pixels, which may include: selecting a preset number of pixels from each of the multiple second sub-images corresponding to each channel of the image of the second original image, obtains at least one group of pixels corresponding to the multiple second sub-images, and determines at least one new second image of this round of amplification based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each second sub-image have not been traversed, performing the steps of selecting a preset number of pixels from each second sub-image, obtains at least one group of pixels corresponding to the multiple second sub-images, and determines at least one new second image of a new round of amplification based on the at least one group of pixels, until all combinations of the preset number of pixels in each first sub-image have been traversed, and a new first image obtained by multiple rounds of amplification obtains multiple new second images.

[0080] For an example of this implementation, please refer to the example in the aforementioned step S304, which will not be described again here.

[0081] S305. Construct an original sample pair based on the first original image and the second original image, and construct multiple augmented sample pairs based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0082] In an embodiment of the present application, a computing device may first acquire a first original image of a sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to a behavior to be identified of the sample object; the first original image and the second original image are both three-channel images; then, the images of the three channels corresponding to the first original image are respectively subjected to image division processing to obtain a plurality of first sub-images corresponding to the image of each channel, and the images of the three channels corresponding to the second original image are respectively subjected to image division processing to obtain a plurality of second sub-images corresponding to the image of each channel; further, for the plurality of first sub-images corresponding to the image of each channel of the first original image, a preset number of pixel points are respectively selected from each first sub-image to obtain a plurality of groups of pixel points corresponding to the plurality of first sub-images, and based on the plurality of groups of pixel points, determine A new first image; a new first image is obtained by amplifying the images of the three channels of the first original image, so as to obtain multiple new first images; and, for the multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image is determined based on the multiple groups of pixels; a new first image is obtained by amplifying the images of the three channels of the second original image, so as to obtain multiple new first images; finally, an original sample pair is constructed based on the first original image and the second original image, and a multiple amplified sample pairs are constructed based on the multiple new first images and the multiple new second images; the original sample pairs and the multiple amplified sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin. By adopting this method, the computing device can obtain a new first image and a new second image by respectively amplifying the first original image and the second original image, and construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image. In this way, when the twin network to be trained is trained using the original sample pair and the amplified sample pair, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training.

[0083] In addition, compared with the method of not using the original sample pairs and amplified sample pairs obtained by the image data processing method to train the twin network to be trained, the use of this method can reduce the number of first original images obtained when the accuracy of the trained twin network is the same.

[0084] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of another image data processing method provided in an embodiment of the present application. Figure 3 The image data processing method shown is different in that Figure 5 In the method shown, the computing device realizes image amplification by exchanging the pixel values ​​of the same pixel points of each channel image in the original image. Figure 5 As shown, the image data processing method may include but is not limited to the following steps:

[0085] S501, obtaining a first original image of a sample object, and determining a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to a behavior to be identified of the sample object; the first original image and the second original image are both three-channel images.

[0086] In an optional implementation, the relevant description of step S501 can refer to the description of the aforementioned step S301, which will not be repeated here.

[0087] S502, for the first original image, update the pixel value of the preset pixel point in the first channel image to the pixel value of the preset pixel point in the second channel image or the third channel image; update the pixel value of the preset pixel point in the second channel image to the pixel value of the preset pixel point in the first channel image or the third channel image; update the pixel value of the preset pixel point in the third channel image to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after the pixel value is updated, the second channel image after the pixel value is updated, and the third channel image after the pixel value is updated, obtain a new first image obtained in this round of amplification; obtain multiple new first images from the new first images obtained by multiple rounds of amplification.

[0088] The following takes the case where the first original image is an RGB image as an example to illustrate step S502.

[0089] See also Figure 6 , Figure 6 FIG. 1 is another schematic diagram of a process for determining a new first image provided by an embodiment of the present application. Figure 6 As shown, the computing device may update the pixel values ​​of the 9 pixels included in area 1 in the R channel image to the pixel values ​​of the 9 pixels included in area 1 in the G channel image, and obtain an updated R channel image, that is, Figure 6 Image 1 in the G channel image; update the pixel values ​​of the 9 pixels in area 1 in the G channel image to the pixel values ​​of the 9 pixels in area 1 in the B channel image, and obtain the updated G channel image, that is, Figure 6 , and the pixel values ​​of the 9 pixels in area 1 of the B channel image are updated to the pixel values ​​of the 9 pixels in area 1 of the R channel image to obtain the updated G channel image, that is, Figure 6Afterwards, the computing device may obtain a new first image obtained in this round of amplification based on image 1, image 2, and image 3.

[0090] In addition, the computing device may also update the pixel values ​​of the 9 pixels included in area 1 in the R channel image to the pixel values ​​of the 9 pixels included in area 1 in the B channel image to obtain an updated R channel image, that is, Figure 6 Image 4 in the G channel image; the pixel values ​​of the 9 pixels in area 1 in the G channel image are updated to the pixel values ​​of the 9 pixels in area 1 in the R channel image to obtain the updated G channel image, that is, Figure 6 5 in the image 5, and the pixel values ​​of the 9 pixels in the region 1 in the B channel image are updated to the pixel values ​​of the 9 pixels in the region 1 in the G channel image to obtain the updated G channel image, that is, Figure 6 Afterwards, the computing device may obtain a new first image obtained in this round of amplification based on image 4, image 5, and image 6.

[0091] S503, for the second original image, updating the pixel value of the preset pixel point in the first channel image to the pixel value of the preset pixel point in the second channel image or the third channel image; updating the pixel value of the preset pixel point in the second channel image to the pixel value of the preset pixel point in the first channel image or the third channel image; updating the pixel value of the preset pixel point in the third channel image to the pixel value of the preset pixel point in the first channel image or the second channel image; obtaining a new second image obtained in this round of amplification based on the first channel image after the pixel value is updated, the second channel image after the pixel value is updated, and the third channel image after the pixel value is updated; obtaining multiple new second images from the new second images obtained by multiple rounds of amplification.

[0092] The example of step S503 can refer to the example of the aforementioned step S304, which will not be described again here.

[0093] S504. Construct an original sample pair based on the first original image and the second original image, and construct multiple augmented sample pairs based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the target object in the vehicle cabin. Behavior to be identified.

[0094] In an embodiment of the present application, the computing device can obtain multiple new first images by exchanging the pixel values ​​of the same pixel points of each channel image of the first original image, and obtain multiple new second images by exchanging the pixel values ​​of the same pixel points of each channel image of the second original image; thereafter, construct an original sample pair based on the first original image and the second original image, and construct an augmented sample pair based on the new first image and the new second image. In this way, when the twin network to be trained is trained using the original sample pairs and the augmented sample pairs, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training.

[0095] In addition, compared with the method of not using the original sample pairs and amplified sample pairs obtained by the image data processing method to train the twin network to be trained, the use of this method can reduce the number of first original images obtained when the accuracy of the trained twin network is the same.

[0096] In an optional embodiment, Figure 2 , Figure 3 as well as Figure 5 In the image data processing method shown, after the computing device constructs an original sample pair based on the first original image and the second original image, and constructs an amplified sample pair based on the new first image and the new second image, it can also train the twin network to be trained based on the original sample pair and the amplified sample pair to obtain a trained twin network.

[0097] In this embodiment, the structure of the twin network can be as follows Figure 7 As shown, Figure 7 Schematic diagram of a twin network structure provided in the embodiment of the present application. Figure 7 As shown, the twin network to be trained may include a first feature extraction network ( Figure 7 The first feature extraction network is a convolutional neural network 1, and the second feature extraction network ( Figure 7 In the figure, the second feature extraction network is a convolutional neural network 2), the first classification network and the second classification network. Optionally, the computing device trains the training twin network based on the original sample pair and the amplified sample pair, which may include but is not limited to the following steps:

[0098] Step 1: Determine a set of sample image pairs based on the original sample pairs and the amplified sample pairs.

[0099] Step 2: For each sample image pair in the sample image pair set, the first sample image in the sample image pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature corresponding to the first sample image, and the second sample image in the sample image pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature corresponding to the second sample image.

[0100] Step three, determine the total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained.

[0101] In an optional implementation, the computing device may use the following formula (1) to determine the total loss value of the sample image pair.

[0102] L = λ×(L cls (G w (D′1)+L cls (G w (D′2)))+(1-2λ)×L rps (1)

[0103] In formula (1), L represents the total loss value of the sample image pair; λ represents the loss weight; G w (D1) is the first feature vector, which represents the first sample image feature; G w (D′2) is the second feature vector, which represents the second sample image feature; L rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; L cls (G w (D1) represents the second loss value, that is, the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object; L cls (G w (D2′) represents the third loss value, that is, the loss value between the second predicted behavior and the actual behavior of the sample object.

[0104] In an optional embodiment, the first loss value (ie, L rps) can be determined by the computing device in the following manner: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0105] In this implementation, the sample image feature similarity between the first sample image feature and the second sample image feature can be determined by the computing device through a difference vector similarity measurement method, or it can be determined by the computing device through a measurement method combining distance similarity and cosine similarity, which is not limited here.

[0106] When the computing device determines the sample image feature similarity between the first sample image feature and the second sample image feature by measuring the difference vector similarity, the following formula (2) may be used.

[0107]

[0108] In formula (2), G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2) is the second feature vector, which represents the second sample image feature; e It means G w The number of elements in (D1′) that are not equal to 0; f It means G w The number of elements in (D2′) that are not equal to 0; ef It means G w (D1′) and G w The number of elements in the difference of (D2′) that are not equal to 0.

[0109] When the computing device determines the sample image feature similarity between the first sample image feature and the second sample image feature by a measurement method combining distance similarity and cosine similarity, the following formula (3) may be used.

[0110] S(G w (D1′), G w (D2′))=dis(G w (D1′), G w (D2′))·cos(G w (D1′), G w (D2′)) (3)

[0111] In formula (3), G w (D1′) is the first feature vector, which represents the first sample image feature; G w(D2′) is the second feature vector, which represents the second sample image feature; dis() represents the distance similarity; and cos() represents the cosine similarity.

[0112] In this implementation, the preset value interval may be [0, 1]. Optionally, the computing device converts the sample image feature similarity into the sample image feature similarity probability within the preset value interval, which may be achieved by using Figure 5 The fully connected layer and sigmoid layer of the recognition network in the twin network to be trained shown convert the sample image feature similarity into a numerical value in the interval [0, 1], namely, RPS, and use RPS as the sample image feature similarity probability.

[0113] It can be understood that if the probability of similarity of sample image features is higher, it means that the recognition result of the first classification network in the twin network for the sample object's to-be-identified behavior in the first sample image (full image) is more consistent with the recognition result of the second classification network for the sample object's to-be-identified behavior in the second sample image (partial image).

[0114] In this implementation, when the computing device determines the first loss value between the first sample image and the second sample image based on the similarity probability of the sample image features, the following formula (4) may be used.

[0115]

[0116] In formula (4), L rps represents the first loss value, that is, the loss value between the first sample image and the second sample image; RPS represents the probability of similarity of sample image features; λ represents the loss weight; 1 represents the label of the sample pair, if the sample pair is a positive sample pair, 1 is 1, if the sample pair is a negative sample pair, 1 is 0; It represents the regularization function.

[0117] Step 4: When it is determined that the total loss value of any sample image pair does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the conditions for stopping training, thereby obtaining the trained twin network.

[0118] With this implementation, on the one hand, the computing device can determine a set of sample image pairs based on the original sample pairs and the augmented sample pairs; and train the twin network to be trained using the set of sample image pairs, so that the twin network can learn more features based on the smallest original image (two original images, i.e., the first original image and the second original image), thereby improving the generalization ability of the twin network after training. On the other hand, the computing device trains the twin network by using the total loss value of each sample image pair in the sample image set, wherein the total loss value of each sample image pair is composed of the first loss value between the first sample image and the second sample image in the sample image pair, the second loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the third loss value between the second predicted behavior and the actual behavior of the sample object; thereafter, when the total loss value of any sample image pair in the sample image pair set does not meet the stop training condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of the multiple sample image pairs meet the stop training condition. In this way, when the target object to be identified in the vehicle cabin is subsequently identified based on the trained twin network, the accuracy of the recognition result can be improved.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides an image data processing device for implementing the image data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image data processing device embodiments provided below can refer to the limitations of the image data processing method above, and will not be repeated here.

[0121] See also Figure 8 , Figure 8 Schematic diagram of the structure of an image data processing device provided in an embodiment of the present application. Figure 8 As shown, the image data processing device may include but is not limited to:

[0122] The acquisition and determination module 801 is used to acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0123] The image amplification module 802 is used to amplify the first original image based on the image of each channel corresponding to the first original image to obtain a plurality of new first images, and to amplify the second original image based on the image of each channel corresponding to the second original image to obtain a plurality of new second images;

[0124] Construction module 803 is used to construct an original sample pair based on the first original image and the second original image, and to construct multiple augmented sample pairs based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0125] In one embodiment, the image amplification module 802 is used to amplify the first original image based on the image of each channel corresponding to the first original image to obtain multiple new first images, and to amplify the second original image based on the image of each channel corresponding to the second original image to obtain multiple new second images, and is specifically used to: perform image division processing on the images of the three channels corresponding to the first original image respectively to obtain multiple first sub-images corresponding to the image of each channel, and perform image division processing on the images of the three channels corresponding to the second original image respectively to obtain multiple second sub-images corresponding to the image of each channel; for the multiple first sub-images corresponding to the image of each channel of the first original image Sub-images, a preset number of pixels are selected from each first sub-image to obtain multiple groups of pixels corresponding to multiple first sub-images, and a new first image is determined based on the multiple groups of pixels; a new first image is obtained by respectively amplifying the images of the three channels of the first original image to obtain multiple new first images; for multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain multiple groups of pixels corresponding to multiple second sub-images, and a new second image is determined based on the multiple groups of pixels; a new first image is obtained by respectively amplifying the images of the three channels of the second original image to obtain multiple new first images.

[0126] In one embodiment, the image augmentation module 802 is used to perform image division processing on the images of the three channels corresponding to the first original image respectively to obtain multiple first sub-images corresponding to the image of each channel, and to perform image division processing on the images of the three channels corresponding to the second original image respectively to obtain multiple second sub-images corresponding to the image of each channel. It is specifically used to: equally divide the pixel points of the images of the three channels corresponding to the first original image respectively to obtain multiple equally divided first sub-images corresponding to the image of each channel; and equally divide the pixel points of the images of the three channels corresponding to the second original image respectively to obtain multiple equally divided second sub-images corresponding to the image of each channel.

[0127] In one embodiment, when the image augmentation module 802 is used to select a preset number of pixels from each of the first sub-images corresponding to each channel of the image of the first original image, obtain multiple groups of pixels corresponding to the multiple first sub-images, and determine a new first image based on the multiple groups of pixels, it is specifically used to: select a preset number of pixels from each of the first sub-images corresponding to each channel of the image of the first original image, obtain at least one group of pixels corresponding to the multiple first sub-images, and determine at least one new first image of this round of augmentation based on the at least one group of pixels; if it is determined that all combinations of the preset number of pixels in each first sub-image have not been traversed, perform the steps of selecting a preset number of pixels from each of the first sub-images, obtain at least one group of pixels corresponding to the multiple first sub-images, and determine at least one new first image of a new round of augmentation based on the at least one group of pixels; until all combinations of the preset number of pixels in each first sub-image have been traversed, and a new first image obtained by multiple rounds of amplification is obtained, so as to obtain multiple new first images.

[0128] In one embodiment, when the image augmentation module 802 is used to select a preset number of pixels from each of the second sub-images corresponding to each channel of the image of the second original image, obtain multiple groups of pixels corresponding to the multiple second sub-images, and determine a new second image based on the multiple groups of pixels, it is specifically used to: select a preset number of pixels from each of the second sub-images corresponding to each channel of the image of the second original image, obtain at least one group of pixels corresponding to the multiple second sub-images, and determine at least one new second image of this round of augmentation based on the at least one group of pixels; if it is determined that all combinations of the preset number of pixels in each second sub-image have not been traversed, perform the steps of selecting a preset number of pixels from each second sub-image, obtain at least one group of pixels corresponding to the multiple second sub-images, and determine at least one new second image of a new round of augmentation based on the at least one group of pixels, until all combinations of the preset number of pixels in each first sub-image have been traversed, and a new first image obtained by multiple rounds of amplification is obtained, thereby obtaining multiple new second images.

[0129] In one embodiment, the image amplification module 802 is used to amplify the first original image based on the image of each channel corresponding to the first original image to obtain multiple new first images, and to amplify the second original image based on the image of each channel corresponding to the second original image to obtain multiple new second images. It is specifically used to: for the first original image, update the pixel value of the preset pixel point in the first channel image to the pixel value of the preset pixel point in the second channel image or the third channel image; update the pixel value of the preset pixel point in the second channel image to the pixel value of the preset pixel point in the first channel image or the third channel image; update the pixel value of the preset pixel point in the third channel image to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after the updated pixel value, the second channel image after the updated pixel value, and more The third channel image after the new pixel value is updated, and the new first image obtained in this round of amplification is obtained; from the new first image obtained by multiple rounds of amplification, multiple new first images are obtained; and, for the second original image, the pixel value of the preset pixel point in the first channel image is updated to the pixel value of the preset pixel point in the second channel image or the third channel image; the pixel value of the preset pixel point in the second channel image is updated to the pixel value of the preset pixel point in the first channel image or the third channel image; the pixel value of the preset pixel point in the third channel image is updated to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after the updated pixel value, the second channel image after the updated pixel value and the third channel image after the updated pixel value, a new second image obtained in this round of amplification is obtained; from the new second image obtained by multiple rounds of amplification, multiple new second images are obtained.

[0130] In one embodiment, the device also includes a training module; the training module is used to determine a set of sample image pairs based on the original sample pairs and multiple amplified sample pairs; for each sample image pair in the sample image pair set, the first sample image in the sample image pair is input into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and the second sample image in the sample image pair is input into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; determine the total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image The feature is determined by the second sample image feature; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the training stop condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the training stop condition to obtain the trained twin network.

[0131] Each module in the above-mentioned image data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computing device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0132] In an exemplary embodiment, a computing device is provided, which can be applied in a vehicle. The computing device can be a terminal, and its internal structure diagram can be as follows: Fig. 9As shown. The computing device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computing device is used to provide computing and control capabilities. The memory of the computing device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computing device is used to exchange information between the processor and an external device. The communication interface of the computing device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, the steps in an image data processing method are implemented. The display unit of the computing device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computing device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computing device shell, or an external keyboard, touchpad, etc.

[0133] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computing device or server to which the solution of the present application is applied. The specific computing device or server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] In an exemplary embodiment, the present application provides a computing device, including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, the following steps are implemented:

[0135] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0136] Based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images; and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images;

[0137] An original sample pair is constructed based on the first original image and the second original image, and multiple augmented sample pairs are constructed based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0138] In one embodiment, when a processor executes a computer program to amplify the first original image based on the image of each channel corresponding to the first original image to obtain multiple new first images, and amplify the second original image based on the image of each channel corresponding to the second original image to obtain multiple new second images, the following steps are specifically implemented: image division processing is performed on the images of the three channels corresponding to the first original image to obtain multiple first sub-images corresponding to the image of each channel, and image division processing is performed on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; for multiple images corresponding to the image of each channel of the first original image, For the first sub-image, a preset number of pixels are selected from each first sub-image to obtain multiple groups of pixels corresponding to the multiple first sub-images, and a new first image is determined based on the multiple groups of pixels; a new first image is obtained by respectively amplifying the images of the three channels of the first original image to obtain multiple new first images; for the multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image is determined based on the multiple groups of pixels; a new first image is obtained by respectively amplifying the images of the three channels of the second original image to obtain multiple new first images.

[0139] In one embodiment, when a processor executes a computer program to implement image division processing on images of three channels corresponding to a first original image, respectively, to obtain a plurality of first sub-images corresponding to the image of each channel, and to implement image division processing on images of three channels corresponding to a second original image, respectively, to obtain a plurality of second sub-images corresponding to the image of each channel, the following steps are specifically implemented: equally dividing the pixel points of the images of the three channels corresponding to the first original image, respectively, to obtain a plurality of equally divided first sub-images corresponding to the image of each channel; and equally dividing the pixel points of the images of the three channels corresponding to the second original image, respectively, to obtain a plurality of equally divided second sub-images corresponding to the image of each channel.

[0140] In one embodiment, when a processor executes a computer program to select a preset number of pixels from each first sub-image for a plurality of first sub-images corresponding to an image of each channel of a first original image, obtain a plurality of groups of pixels corresponding to the plurality of first sub-images, and determine a new first image based on the plurality of groups of pixels, the following steps are specifically implemented: for a plurality of first sub-images corresponding to an image of each channel of the first original image, select a preset number of pixels from each first sub-image, obtain at least one group of pixels corresponding to the plurality of first sub-images, and determine at least one new first image of this round of amplification based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each first sub-image have not been traversed, perform the steps of selecting a preset number of pixels from each first sub-image, obtain at least one group of pixels corresponding to the plurality of first sub-images, and determine at least one new first image of a new round of amplification based on the at least one group of pixels; until all combinations of the preset number of pixels in each first sub-image have been traversed, and a plurality of new first images are obtained by multiple rounds of amplification.

[0141] In one embodiment, when a processor executes a computer program to implement, for a plurality of second sub-images corresponding to an image of each channel of the second original image, respectively selecting a preset number of pixels from each second sub-image to obtain a plurality of groups of pixels corresponding to the plurality of second sub-images, and determining a new second image based on the plurality of groups of pixels, the following steps are specifically implemented: for a plurality of second sub-images corresponding to an image of each channel of the second original image, respectively selecting a preset number of pixels from each second sub-image to obtain at least one group of pixels corresponding to the plurality of second sub-images, and determining at least one new second image of this round of amplification based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each second sub-image have not been traversed, executing the steps of respectively selecting a preset number of pixels from each second sub-image to obtain at least one group of pixels corresponding to the plurality of second sub-images, and determining at least one new second image of a new round of amplification based on the at least one group of pixels, until all combinations of the preset number of pixels in each first sub-image have been traversed, and a plurality of new second images are obtained by obtaining a new first image obtained by multiple rounds of amplification.

[0142] In one embodiment, when a processor executes a computer program to amplify the first original image based on the image of each channel corresponding to the first original image to obtain multiple new first images, and to amplify the second original image based on the image of each channel corresponding to the second original image to obtain multiple new second images, the following steps are specifically implemented: for the first original image, the pixel value of a preset pixel point in the first channel image is updated to the pixel value of a preset pixel point in the second channel image or the third channel image; the pixel value of a preset pixel point in the second channel image is updated to the pixel value of a preset pixel point in the first channel image or the third channel image; the pixel value of a preset pixel point in the third channel image is updated to the pixel value of a preset pixel point in the first channel image or the second channel image; based on the first channel image after the pixel value is updated and the second channel image after the pixel value is updated, and the third channel image after updating the pixel value, to obtain a new first image obtained in this round of amplification; from the new first image obtained by multiple rounds of amplification, multiple new first images are obtained; and, for the second original image, the pixel value of the preset pixel point in the first channel image is updated to the pixel value of the preset pixel point in the second channel image or the third channel image; the pixel value of the preset pixel point in the second channel image is updated to the pixel value of the preset pixel point in the first channel image or the third channel image; the pixel value of the preset pixel point in the third channel image is updated to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after updating the pixel value, the second channel image after updating the pixel value and the third channel image after updating the pixel value, a new second image obtained in this round of amplification is obtained; from the new second image obtained by multiple rounds of amplification, multiple new second images are obtained.

[0143] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the original sample pairs and the multiple amplified sample pairs, a set of sample image pairs is determined; for each sample image pair in the set of sample image pairs, the first sample image in the sample image pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature corresponding to the first sample image, and the second sample image in the sample image pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature corresponding to the second sample image; the total loss value of the sample image pair is determined; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image The feature is determined by the second sample image feature; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the training stop condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the training stop condition to obtain the trained twin network.

[0144] In an exemplary embodiment, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0145] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0146] Based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images; and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images;

[0147] An original sample pair is constructed based on the first original image and the second original image, and multiple augmented sample pairs are constructed based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0148] In one embodiment, the computer program is executed by a processor to implement the following steps: performing image division processing on the images of the three channels corresponding to the first original image to obtain multiple first sub-images corresponding to the image of each channel, and performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the first ... first sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the first original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the first original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of A first sub-image is obtained, a preset number of pixels are selected from each first sub-image to obtain multiple groups of pixels corresponding to the multiple first sub-images, and a new first image is determined based on the multiple groups of pixels; a new first image is obtained by respectively amplifying the images of the three channels of the first original image to obtain multiple new first images; for multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image is determined based on the multiple groups of pixels; a new first image is obtained by respectively amplifying the images of the three channels of the second original image to obtain multiple new first images.

[0149] In one embodiment, a computer program is executed by a processor to implement image division processing on images of three channels corresponding to a first original image, respectively, to obtain a plurality of first sub-images corresponding to the image of each channel, and to implement image division processing on images of three channels corresponding to a second original image, respectively, to obtain a plurality of second sub-images corresponding to the image of each channel. Specifically, the following steps are implemented: pixel points of the images of the three channels corresponding to the first original image are equally divided, respectively, to obtain a plurality of equally divided first sub-images corresponding to the image of each channel; and pixel points of the images of the three channels corresponding to the second original image are equally divided, respectively, to obtain a plurality of equally divided second sub-images corresponding to the image of each channel.

[0150] In one embodiment, when a computer program is executed by a processor to implement, for a plurality of first sub-images corresponding to an image of each channel of a first original image, respectively selecting a preset number of pixels from each first sub-image to obtain a plurality of groups of pixels corresponding to the plurality of first sub-images, and determining a new first image based on the plurality of groups of pixels, the following steps are specifically implemented: for a plurality of first sub-images corresponding to an image of each channel of the first original image, respectively selecting a preset number of pixels from each first sub-image to obtain at least one group of pixels corresponding to the plurality of first sub-images, and determining at least one new first image of this round of amplification based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each first sub-image have not been traversed, executing the steps of respectively selecting a preset number of pixels from each first sub-image to obtain at least one group of pixels corresponding to the plurality of first sub-images, and determining at least one new first image of a new round of amplification based on the at least one group of pixels; until all combinations of the preset number of pixels in each first sub-image have been traversed, and a plurality of new first images are obtained by obtaining new first images obtained by multiple rounds of amplification.

[0151] In one embodiment, when a computer program is executed by a processor to implement, for a plurality of second sub-images corresponding to an image of each channel of a second original image, respectively selecting a preset number of pixels from each second sub-image to obtain a plurality of groups of pixels corresponding to the plurality of second sub-images, and determining a new second image based on the plurality of groups of pixels, the following steps are specifically implemented: for a plurality of second sub-images corresponding to an image of each channel of the second original image, respectively selecting a preset number of pixels from each second sub-image to obtain at least one group of pixels corresponding to the plurality of second sub-images, and determining at least one new second image of this round of amplification based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each second sub-image have not been traversed, executing the steps of respectively selecting a preset number of pixels from each second sub-image to obtain at least one group of pixels corresponding to the plurality of second sub-images, and determining at least one new second image of a new round of amplification based on the at least one group of pixels, until all combinations of the preset number of pixels in each first sub-image have been traversed, and a plurality of new second images are obtained by obtaining a new first image obtained by multiple rounds of amplification.

[0152] In one embodiment, the computer program is executed by a processor to implement the amplification of the first original image based on the image of each channel corresponding to the first original image to obtain multiple new first images, and the amplification of the second original image based on the image of each channel corresponding to the second original image to obtain multiple new second images, and specifically implement the following steps: for the first original image, update the pixel value of the preset pixel point in the first channel image to the pixel value of the preset pixel point in the second channel image or the third channel image; update the pixel value of the preset pixel point in the second channel image to the pixel value of the preset pixel point in the first channel image or the third channel image; update the pixel value of the preset pixel point in the third channel image to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after the pixel value is updated and the second channel image after the pixel value is updated and the third channel image after updating the pixel value, to obtain a new first image obtained in this round of amplification; the new first image obtained by multiple rounds of amplification obtains multiple new first images; and, for the second original image, the pixel value of the preset pixel point in the first channel image is updated to the pixel value of the preset pixel point in the second channel image or the third channel image; the pixel value of the preset pixel point in the second channel image is updated to the pixel value of the preset pixel point in the first channel image or the third channel image; the pixel value of the preset pixel point in the third channel image is updated to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after updating the pixel value, the second channel image after updating the pixel value and the third channel image after updating the pixel value, obtain the new second image obtained in this round of amplification; the new second image obtained by multiple rounds of amplification obtains multiple new second images.

[0153] In one embodiment, the computer program is executed by the processor to further implement the following steps: based on the original sample pairs and the multiple amplified sample pairs, determine a set of sample image pairs; for each sample image pair in the set of sample image pairs, input the first sample image in the sample image pair into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and input the second sample image in the sample image pair into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; determine the total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image The feature is determined by the second sample image feature; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the training stop condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the training stop condition to obtain the trained twin network.

[0154] In an exemplary embodiment, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0155] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images;

[0156] Based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images; and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images;

[0157] An original sample pair is constructed based on the first original image and the second original image, and multiple augmented sample pairs are constructed based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0158] In one embodiment, the computer program is executed by a processor to implement the following steps: performing image division processing on the images of the three channels corresponding to the first original image to obtain multiple first sub-images corresponding to the image of each channel, and performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the first ... first sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the first original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the first original image to obtain multiple second sub-images corresponding to the image of each channel; performing image division processing on the images of the three channels corresponding to the second original image to obtain multiple second sub-images corresponding to the image of A first sub-image is obtained, a preset number of pixels are selected from each first sub-image to obtain multiple groups of pixels corresponding to the multiple first sub-images, and a new first image is determined based on the multiple groups of pixels; a new first image is obtained by respectively amplifying the images of the three channels of the first original image to obtain multiple new first images; for multiple second sub-images corresponding to the image of each channel of the second original image, a preset number of pixels are selected from each second sub-image to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image is determined based on the multiple groups of pixels; a new first image is obtained by respectively amplifying the images of the three channels of the second original image to obtain multiple new first images.

[0159] In one embodiment, a computer program is executed by a processor to implement image division processing on images of three channels corresponding to a first original image, respectively, to obtain a plurality of first sub-images corresponding to the image of each channel, and to implement image division processing on images of three channels corresponding to a second original image, respectively, to obtain a plurality of second sub-images corresponding to the image of each channel. Specifically, the following steps are implemented: pixel points of the images of the three channels corresponding to the first original image are equally divided, respectively, to obtain a plurality of equally divided first sub-images corresponding to the image of each channel; and pixel points of the images of the three channels corresponding to the second original image are equally divided, respectively, to obtain a plurality of equally divided second sub-images corresponding to the image of each channel.

[0160] In one embodiment, when a computer program is executed by a processor to implement, for a plurality of first sub-images corresponding to an image of each channel of a first original image, respectively selecting a preset number of pixels from each first sub-image to obtain a plurality of groups of pixels corresponding to the plurality of first sub-images, and determining a new first image based on the plurality of groups of pixels, the following steps are specifically implemented: for a plurality of first sub-images corresponding to an image of each channel of the first original image, respectively selecting a preset number of pixels from each first sub-image to obtain at least one group of pixels corresponding to the plurality of first sub-images, and determining at least one new first image of this round of amplification based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each first sub-image have not been traversed, executing the steps of respectively selecting a preset number of pixels from each first sub-image to obtain at least one group of pixels corresponding to the plurality of first sub-images, and determining at least one new first image of a new round of amplification based on the at least one group of pixels; until all combinations of the preset number of pixels in each first sub-image have been traversed, and a plurality of new first images are obtained by obtaining new first images obtained by multiple rounds of amplification.

[0161] In one embodiment, when a computer program is executed by a processor to implement, for a plurality of second sub-images corresponding to an image of each channel of a second original image, respectively selecting a preset number of pixels from each second sub-image to obtain a plurality of groups of pixels corresponding to the plurality of second sub-images, and determining a new second image based on the plurality of groups of pixels, the following steps are specifically implemented: for a plurality of second sub-images corresponding to an image of each channel of the second original image, respectively selecting a preset number of pixels from each second sub-image to obtain at least one group of pixels corresponding to the plurality of second sub-images, and determining at least one new second image of this round of amplification based on the at least one group of pixels; when it is determined that all combinations of the preset number of pixels in each second sub-image have not been traversed, executing the steps of respectively selecting a preset number of pixels from each second sub-image to obtain at least one group of pixels corresponding to the plurality of second sub-images, and determining at least one new second image of a new round of amplification based on the at least one group of pixels, until all combinations of the preset number of pixels in each first sub-image have been traversed, and a plurality of new second images are obtained by obtaining a new first image obtained by multiple rounds of amplification.

[0162] In one embodiment, the computer program is executed by a processor to implement the amplification of the first original image based on the image of each channel corresponding to the first original image to obtain multiple new first images, and the amplification of the second original image based on the image of each channel corresponding to the second original image to obtain multiple new second images, and specifically implement the following steps: for the first original image, update the pixel value of the preset pixel point in the first channel image to the pixel value of the preset pixel point in the second channel image or the third channel image; update the pixel value of the preset pixel point in the second channel image to the pixel value of the preset pixel point in the first channel image or the third channel image; update the pixel value of the preset pixel point in the third channel image to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after the pixel value is updated and the second channel image after the pixel value is updated and the third channel image after updating the pixel value, to obtain a new first image obtained in this round of amplification; the new first image obtained by multiple rounds of amplification obtains multiple new first images; and, for the second original image, the pixel value of the preset pixel point in the first channel image is updated to the pixel value of the preset pixel point in the second channel image or the third channel image; the pixel value of the preset pixel point in the second channel image is updated to the pixel value of the preset pixel point in the first channel image or the third channel image; the pixel value of the preset pixel point in the third channel image is updated to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after updating the pixel value, the second channel image after updating the pixel value and the third channel image after updating the pixel value, obtain the new second image obtained in this round of amplification; the new second image obtained by multiple rounds of amplification obtains multiple new second images.

[0163] In one embodiment, the computer program is executed by the processor to further implement the following steps: based on the original sample pairs and the multiple amplified sample pairs, determine a set of sample image pairs; for each sample image pair in the set of sample image pairs, input the first sample image in the sample image pair into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and input the second sample image in the sample image pair into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; determine the total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image The feature is determined by the second sample image feature; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the training stop condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the training stop condition to obtain the trained twin network.

[0164] It should be noted that the data involved in this application (including but not limited to the first original image of the sample object, the second original image of the sample object, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations. In addition, the new first image, the new second image, the original sample pair and the amplified sample pair involved in this application are also images authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant images must comply with relevant regulations.

[0165] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0166] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0167] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing image data, characterized in that: The method comprises: Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images; Based on the image of each channel corresponding to the first original image, the first original image is amplified to obtain a plurality of new first images; and based on the image of each channel corresponding to the second original image, the second original image is amplified to obtain a plurality of new second images; An original sample pair is constructed based on the first original image and the second original image, and a plurality of augmented sample pairs are constructed based on a plurality of the new first images and a plurality of the new second images; the original sample pair and the plurality of the augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

2. The method according to claim 1, characterized in that The method of amplifying the first original image based on the image of each channel corresponding to the first original image to obtain a plurality of new first images, and amplifying the second original image based on the image of each channel corresponding to the second original image to obtain a plurality of new second images, comprises: Performing image division processing on the images of the three channels corresponding to the first original image respectively to obtain a plurality of first sub-images corresponding to the image of each channel, and performing image division processing on the images of the three channels corresponding to the second original image respectively to obtain a plurality of second sub-images corresponding to the image of each channel; For a plurality of first sub-images corresponding to the image of each of the channels of the first original image, a preset number of pixel points are respectively selected from each of the first sub-images to obtain a plurality of groups of pixel points corresponding to the plurality of first sub-images, and a new first image is determined based on the plurality of groups of pixel points; based on the new first images respectively amplified by the images of the three channels of the first original image, a plurality of new first images are obtained; For the multiple second sub-images corresponding to the image of each of the channels of the second original image, the preset number of pixel points are selected from each of the second sub-images to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and new second images are determined based on the multiple groups of pixel points; based on the images of the three channels of the second original image, new first images are obtained by amplifying the images of the three channels respectively to obtain multiple new first images.

3. The method according to claim 2, characterized in that The image division processing is performed on the images of the three channels corresponding to the first original image to obtain a plurality of first sub-images corresponding to the image of each channel, and the image division processing is performed on the images of the three channels corresponding to the second original image to obtain a plurality of second sub-images corresponding to the image of each channel, including: The pixel points of the images of the three channels corresponding to the first original image are equally divided respectively to obtain a plurality of equally divided first sub-images corresponding to the image of each channel; and The pixel points of the images of the three channels corresponding to the second original image are equally divided respectively to obtain a plurality of equally divided second sub-images corresponding to the image of each channel.

4. The method according to claim 2, characterized in that: The method of selecting a preset number of pixel points from each of the first sub-images corresponding to the image of each channel of the first original image to obtain a plurality of groups of pixel points corresponding to the plurality of first sub-images, and determining a new first image based on the plurality of groups of pixel points includes: For a plurality of first sub-images corresponding to the image of each channel of the first original image, a preset number of pixel points are respectively selected from each of the first sub-images to obtain at least one group of pixel points corresponding to the plurality of first sub-images, and at least one new first image of this round of amplification is determined based on the at least one group of pixel points; When it is determined that all combinations of the preset number of pixel points in each of the first sub-images have not been traversed, performing the steps of selecting the preset number of pixel points from each of the first sub-images to obtain at least one group of pixel points corresponding to the plurality of first sub-images, and determining at least one new first image of a new round of enlargement based on the at least one group of pixel points; Until all combinations of a preset number of pixel points in each of the first sub-images are traversed, a new first image is obtained by multiple rounds of amplification to obtain multiple new first images.

5. The method according to claim 4, characterized in that The method of selecting the preset number of pixels from each of the second sub-images corresponding to the image of each channel of the second original image to obtain a plurality of groups of pixels corresponding to the plurality of second sub-images, and determining a new second image based on the plurality of groups of pixels includes: For the plurality of second sub-images corresponding to the image of each channel of the second original image, a preset number of pixel points are respectively selected from each of the second sub-images to obtain at least one group of pixel points corresponding to the plurality of second sub-images, and at least one new second image of this round of amplification is determined based on the at least one group of pixel points; When it is determined that all combinations of a preset number of pixel points in each of the second sub-images have not been traversed, a step is performed of selecting a preset number of pixel points from each of the second sub-images to obtain at least one group of pixel points corresponding to the multiple second sub-images, and based on the at least one group of pixel points, determining at least one new second image of a new round of amplification, until all combinations of the preset number of pixel points in each of the first sub-images have been traversed, and a new first image is obtained by multiple rounds of amplification, thereby obtaining multiple new second images.

6. The method according to claim 1, characterized in that The method of amplifying the first original image based on the image of each channel corresponding to the first original image to obtain a plurality of new first images, and amplifying the second original image based on the image of each channel corresponding to the second original image to obtain a plurality of new second images, comprises: For the first original image, update the pixel value of the preset pixel point in the first channel image to the pixel value of the preset pixel point in the second channel image or the third channel image; update the pixel value of the preset pixel point in the second channel image to the pixel value of the preset pixel point in the first channel image or the third channel image; update the pixel value of the preset pixel point in the third channel image to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after the pixel value is updated, the second channel image after the pixel value is updated, and the third channel image after the pixel value is updated, obtain a new first image obtained in this round of amplification; obtain multiple new first images from the new first images obtained by multiple rounds of amplification; and, For the second original image, update the pixel value of the preset pixel point in the first channel image to the pixel value of the preset pixel point in the second channel image or the third channel image; update the pixel value of the preset pixel point in the second channel image to the pixel value of the preset pixel point in the first channel image or the third channel image; update the pixel value of the preset pixel point in the third channel image to the pixel value of the preset pixel point in the first channel image or the second channel image; based on the first channel image after the pixel value is updated, the second channel image after the pixel value is updated, and the third channel image after the pixel value is updated, obtain a new second image obtained in this round of amplification; obtain multiple new second images from the new second images obtained by multiple rounds of amplification.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determine a set of sample image pairs based on the original sample pair and the plurality of amplified sample pairs; For each sample image pair in the sample image pair set, input the first sample image in the sample image pair into the first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and input the second sample image in the sample image pair into the second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; Determine the total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is a loss value between a first predicted behavior of the sample object and an actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into a first classification network included in the twin network to be trained; the third loss value is a loss value between a second predicted behavior of the sample object and the actual behavior, and the second predicted behavior is obtained by inputting the second sample image feature into a second classification network included in the twin network to be trained; When it is determined that the total loss value of any of the sample image pairs does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each of the sample image pairs, and the twin network to be trained is trained again until the total loss values ​​of the multiple sample image pairs respectively meet the condition for stopping training, thereby obtaining the trained twin network.

8. An image data processing device, characterized in that: The device comprises: an acquisition and determination module, configured to acquire a first original image of a sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; the first original image and the second original image are both three-channel images; an image amplification module, configured to amplify the first original image based on an image of each channel corresponding to the first original image to obtain a plurality of new first images, and to amplify the second original image based on an image of each channel corresponding to the second original image to obtain a plurality of new second images; A construction module is used to construct an original sample pair based on the first original image and the second original image, and to construct a plurality of augmented sample pairs based on a plurality of the new first images and a plurality of the new second images; the original sample pair and the plurality of the augmented sample pairs are used together to train a twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

9. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images