Image data processing method and device, equipment and storage medium

Abstract

The invention discloses an image data processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring a first original image of a sample object, and determining a second original image based on the first original image; determining a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image; selecting a preset number of pixel points from each first sub-image, and determining a new first image based on multiple groups of pixel points corresponding to the plurality of first sub-images; selecting a preset number of pixel points from each second sub-image, and determining a new second image based on a plurality of groups of pixel points corresponding to the plurality of second sub-images; constructing an original sample pair based on the first original image and the second original image, and constructing an amplified sample pair based on the new first image and the new second image; the original sample pair and the amplified sample pair are jointly used for training the twin network to be trained. In this way, the generalization ability of the trained twin network can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to an image data processing method, device, equipment and storage medium. Background Art

[0002] At present, the twin network used to identify the behavior of the target object in the vehicle cabin is trained in the following way: face detection is performed on the main driver's seat image frame sequence. If a face is detected, the face key point regression is performed in the face region of interest (ROI); the image of the ROI area is intercepted according to the key points; then, the intercepted image of the ROI area is input into the twin network to be trained, and the twin network to be trained is trained to obtain the twin network used to identify the behavior of the target object in the vehicle cabin. In this way, due to the small amount of data input into the twin network to be trained, the twin network learns fewer features, which in turn leads to poor generalization ability of the trained twin network.

[0003] Therefore, how to improve the generalization ability of the trained twin network has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide an image data processing method, apparatus, device and storage medium, which can improve the generalization ability of a trained twin network.

[0005] In a first aspect, an embodiment of the present application provides an image data processing method, the method comprising:

[0006] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0007] By pixel point division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained;

[0008] Selecting a preset number of pixel points from each first sub-image respectively to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determining a new first image based on the multiple groups of pixel points;

[0009] Selecting a preset number of pixel points from each second sub-image to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points;

[0010] An original sample pair is constructed based on the first original image and the second original image, and an augmented sample pair is constructed based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0011] In one embodiment, a preset number of pixel points are selected from each first sub-image to obtain multiple groups of pixel points corresponding to multiple first sub-images, and a new first image is determined based on the multiple groups of pixel points, including: selecting a preset number of pixel points from each first sub-image to obtain at least one group of pixel points corresponding to multiple first sub-images, and determining at least one new first image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each first sub-image have not been traversed, executing the step of selecting a preset number of pixel points from each first sub-image to obtain at least one group of pixel points corresponding to multiple first sub-images, and determining at least one new first image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new first images are obtained.

[0012] In one embodiment, a preset number of pixel points are selected from each second sub-image to obtain multiple groups of pixel points corresponding to multiple second sub-images, and a new second image is determined based on the multiple groups of pixel points, including: selecting a preset number of pixel points from each second sub-image to obtain at least one group of pixel points corresponding to multiple second sub-images, and determining at least one new second image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each second sub-image have not been traversed, executing the step of selecting a preset number of pixel points from each second sub-image to obtain at least one group of pixel points corresponding to multiple second sub-images, and determining at least one new second image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image are traversed, a new first image is obtained by multiple rounds of amplification, and multiple new second images are obtained.

[0013] In one embodiment, determining a second original image based on a first original image includes: inputting the first original image into a face detection model, and when detecting that the first original image contains a sample object, outputting face information corresponding to the sample object; based on the face information, capturing a face image corresponding to the sample object from the first original image; inputting the face image into a face key point detection model to obtain the face key point coordinates of the sample object; determining a target area based on the face key point coordinates, and obtaining a second original image corresponding to the target area from the face image.

[0014] In one embodiment, multiple first sub-images corresponding to the first original image and multiple second sub-images corresponding to the second original image are obtained by pixel point division, including: equally dividing the pixel points of the first original image to obtain multiple first sub-images corresponding to the first original image after equal division; and equally dividing the pixel points of the second original image to obtain multiple second sub-images corresponding to the second original image after equal division.

[0015] In one embodiment, the method further includes: determining a set of sample image pairs based on the original sample pairs and the amplified sample pairs; for each sample image pair in the set of sample image pairs, inputting the first sample image in the sample image pair into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and inputting the second sample image in the sample image pair into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; determining a total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image feature and the second sample image feature. The image feature determines the loss value; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the training stop condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the training stop condition to obtain the trained twin network.

[0016] In one embodiment, the first loss value is determined by: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining the first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0017] In a second aspect, an embodiment of the present application provides an image data processing device, the device comprising:

[0018] An acquisition and determination module, used to acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0019] An image division module, used for obtaining a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image by dividing the pixels;

[0020] A determination module, configured to select a preset number of pixel points from each first sub-image, obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determine a new first image based on the multiple groups of pixel points;

[0021] The determination module is further used to select a preset number of pixel points from each second sub-image, obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determine a new second image based on the multiple groups of pixel points;

[0022] A construction module is used to construct an original sample pair based on the first original image and the second original image, and to construct an augmented sample pair based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the target object's behavior to be identified in the vehicle cabin.

[0023] In a third aspect, an embodiment of the present application provides a computing device, including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, the following steps are implemented:

[0024] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0025] By pixel point division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained;

[0026] Selecting a preset number of pixel points from each first sub-image respectively to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determining a new first image based on the multiple groups of pixel points;

[0027] Selecting a preset number of pixel points from each second sub-image to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points;

[0028] An original sample pair is constructed based on the first original image and the second original image, and an amplified sample pair is constructed based on the new first image and the new second image; the original sample pair and the amplified sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0030] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0031] By pixel point division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained;

[0032] Selecting a preset number of pixel points from each first sub-image respectively to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determining a new first image based on the multiple groups of pixel points;

[0033] Selecting a preset number of pixel points from each second sub-image to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points;

[0034] An original sample pair is constructed based on the first original image and the second original image, and an amplified sample pair is constructed based on the new first image and the new second image; the original sample pair and the amplified sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0035] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0036] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0037] By pixel point division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained;

[0038] Selecting a preset number of pixel points from each first sub-image respectively to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determining a new first image based on the multiple groups of pixel points;

[0039] Selecting a preset number of pixel points from each second sub-image to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points;

[0040] An original sample pair is constructed based on the first original image and the second original image, and an augmented sample pair is constructed based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0041] The above-mentioned image data processing method, device, vehicle and storage medium, the computing device can obtain the first original image of the sample object, and determine the second original image based on the first original image; the second original image is a local image corresponding to the target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; through pixel point division, multiple first sub-images corresponding to the first original image and multiple second sub-images corresponding to the second original image are obtained; a preset number of pixels are selected from each first sub-image respectively to obtain multiple groups of pixels corresponding to the multiple first sub-images, and a new first image is determined based on the multiple groups of pixels; a preset number of pixels are selected from each second sub-image respectively to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image is determined based on the multiple groups of pixels; an original sample pair is constructed based on the first original image and the second original image, and an augmented sample pair is constructed based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin. By adopting this method, the computing device can obtain a new first image and a new second image by respectively amplifying the first original image and the second original image, and construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image. In this way, when the twin network to be trained is trained using the original sample pair and the amplified sample pair, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a schematic diagram of an application scenario of an image data processing method provided in an embodiment of the present application;

[0044] Figure 2 It is a flowchart of an image data processing method provided in an embodiment of the present application;

[0045] Figure 3 is a flowchart of another image data processing method provided in an embodiment of the present application;

[0046] Figure 4a is a schematic diagram of image division provided in an embodiment of the present application;

[0047] Figure 4b is a schematic diagram of a process for determining a new first image provided by an embodiment of the present application;

[0048] Figure 4c is a schematic diagram of another process of determining a new first image provided by an embodiment of the present application;

[0049] Figure 4d is another schematic diagram of a process for determining a new first image provided by an embodiment of the present application;

[0050] Figure 5 It is a structural diagram of a twin network provided in an embodiment of the present application;

[0051] Figure 6 is a structural schematic diagram of an image data processing device provided in an embodiment of the present application;

[0052] Figure 7 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] First, combine Figure 1 The application scenarios of the image data processing method provided in the embodiment of the present application are introduced. Figure 1, Figure 1 Schematic diagram of an application scenario of an image data processing method provided in an embodiment of the present application. Figure 1 As shown, the application scenario includes a computing device 101 and a database server 102 .

[0055] The computing device 101 may first obtain a first original image of the sample object from the database server 102, and then the computing device 101 may determine a second original image based on the first original image, that is, obtain a local image of the sample object from the global image of the sample object; further, the computing device 101 may divide the first original image into multiple first sub-images and the second original image into multiple second sub-images by pixel point division; then, a preset number of pixels are selected from each first sub-image respectively to obtain multiple groups of pixels corresponding to the multiple first sub-images, and a new first image composed of the multiple groups of pixels is determined; and a preset number of pixels are selected from each second sub-image respectively to obtain multiple groups of pixels corresponding to the multiple second sub-images, and a new second image composed of the multiple groups of pixels is determined; finally, the computing device 101 may construct an original sample pair based on the first original image and the second original image, and construct an augmented sample pair based on the new first image and the new second image; wherein the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin. By adopting this method, the computing device can obtain a new first image and a new second image by respectively amplifying the first original image and the second original image, and construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image. In this way, when the twin network to be trained is trained using the original sample pair and the amplified sample pair, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training.

[0056] Optionally, the computing device 101 may be a terminal device or a server. The terminal devices mentioned here may include but are not limited to: smart phones, tablet computers, laptop computers, desktop computers, smart watches, smart TVs, smart car terminals, etc. The server mentioned here may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers.

[0057] The following is a detailed description of the image data processing method provided in the embodiment of the present application.

[0058] See also Figure 2 , Figure 2is a flow chart of an image data processing method provided by an embodiment of the present application. The method can be performed by a vehicle (e.g. Figure 1 The computing device 101 in the embodiment is executed. Figure 2 As shown, the image data processing method may include but is not limited to the following steps:

[0059] S201, obtaining a first original image of a sample object, and determining a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to a behavior to be identified of the sample object.

[0060] The sample object may be a driver in a vehicle cabin, and correspondingly, the first original image of the sample object is the first original image of the driver. Assuming that the behavior to be identified is whether the driver smokes, the target area is the area related to the smoking behavior, that is, the driver's mouth area. In this case, the second original image is a partial image corresponding to the mouth area in the first original image of the driver.

[0061] In the case where the sample object is an object in the vehicle cabin, before step S201, the computing device may also control the image acquisition device in the cabin to acquire an image of the sample object, and then process the acquired image to obtain a first original image of the sample object, and store the first original image in the database server. The image acquisition device is, for example, an infrared sensor. The image of the sample object may be a single-channel infrared image of the sample object in the cabin acquired by the infrared sensor. In this case, the computing device may convert the single-channel infrared image into a three-channel infrared image, and use the three-channel infrared image as the first original image of the sample object; or the single-channel infrared image may be directly used as the first original image of the sample object without processing the single-channel infrared image.

[0062] S202 . Obtain a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image by pixel point division.

[0063] That is, the computing device may divide the first original image based on the pixels of the first original image to obtain a plurality of first sub-images, and divide the second original image based on the pixels of the second original image to obtain a plurality of second sub-images.

[0064] For example, assuming that the first original image is a 6*6 image, the computing device may divide the first original image into four 3*3 first sub-images based on the pixels of the first original image. For another example, assuming that the first original image is an 8*8 image, the computing device may divide the first original image into four 4*4 first sub-images based on the pixels of the first original image.

[0065] S203 , selecting a preset number of pixel points from each first sub-image to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determining a new first image based on the multiple groups of pixel points.

[0066] Optionally, the preset number may be at least one of the following: 1, 2, 3, ...

[0067] For example, assuming that the first sub-image is a 3*3 image, the preset number may be at least one of the following: 1, 2, 3. That is, the computing device may select 1 pixel from each 3*3 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or, select 2 pixel points from each 3*3 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or, select 2 pixel points from each 3*3 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points.

[0068] For another example, assuming that the first sub-image is a 4*4 image, the preset number may be at least one of the following: 1, 2, 3, 4. That is, the computing device may select 1 pixel from each 4*4 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or, select 2 pixel points from each 4*4 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or, select 2 pixel points from each 4*4 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or, select 4 pixel points from each 4*4 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points.

[0069] S204 , selecting a preset number of pixel points from each second sub-image to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points.

[0070] In an optional implementation, the relevant description of step S204 can refer to the description of the aforementioned step S203, which will not be repeated here.

[0071] S205. Construct an original sample pair based on the first original image and the second original image, and construct an augmented sample pair based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0072] In an embodiment of the present application, a computing device may obtain a first original image of a sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; through pixel point division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained; a preset number of pixels are selected from each first sub-image respectively to obtain a plurality of groups of pixels corresponding to the plurality of first sub-images, and a new first image is determined based on the plurality of groups of pixels; a preset number of pixels are selected from each second sub-image respectively to obtain a plurality of groups of pixels corresponding to the plurality of second sub-images, and a new second image is determined based on the plurality of groups of pixels; an original sample pair is constructed based on the first original image and the second original image, and an augmented sample pair is constructed based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train a twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin. By adopting this method, the computing device can obtain a new first image and a new second image by respectively amplifying the first original image and the second original image, and construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image. In this way, when the twin network to be trained is trained using the original sample pair and the amplified sample pair, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training.

[0073] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another image data processing method provided in an embodiment of the present application. Figure 2 The image data processing method shown is different in that Figure 3 The method shown specifically explains how the computing device selects a preset number of pixel points from each first sub-image, obtains multiple groups of pixel points corresponding to multiple first sub-images, and determines a new first image based on the multiple groups of pixel points, and how to select a preset number of pixel points from each second sub-image, obtains multiple groups of pixel points corresponding to multiple second sub-images, and determines a new second image based on the multiple groups of pixel points.

[0074] like Figure 3 As shown, the image data processing method may include but is not limited to the following steps:

[0075] S301, obtaining a first original image of a sample object, and determining a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to a behavior to be identified of the sample object.

[0076] In an optional embodiment, the computing device determines the second original image based on the first original image, which may include: inputting the first original image into a face detection model, and when it is detected that the first original image contains a sample object, outputting face information corresponding to the sample object; based on the face information, intercepting a face image corresponding to the sample object from the first original image; inputting the face image into a face key point detection model to obtain the face key point coordinates of the sample object; determining a target area based on the face key point coordinates, and obtaining a second original image corresponding to the target area from the face image.

[0077] In this implementation, the face detection model can be a deep learning model or a machine learning model, which is not limited here.

[0078] In this implementation, the face information may include but is not limited to face coordinates, face height, and face width. The computing device intercepts the face image corresponding to the target object from the first image based on the face information, and may intercept the face image corresponding to the target object from the first image based on the face coordinates, face height, and face width.

[0079] In an optional implementation, other related descriptions of step S301 can refer to the description of the aforementioned step S201, which will not be repeated here.

[0080] S302 . Obtain a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image by pixel point division.

[0081] In an optional embodiment, the computing device obtains multiple first sub-images corresponding to the first original image and multiple second sub-images corresponding to the second original image by dividing the pixels, which may include: equally dividing the pixels of the first original image to obtain multiple first sub-images corresponding to the first original image; and equally dividing the pixels of the second original image to obtain multiple second sub-images corresponding to the second original image.

[0082] For example, see Figure 4a , Figure 4a is a schematic diagram of image division provided by an embodiment of the present application. Figure 4aAs shown, the first original image is a 6*6 image. The computing device may divide the first original image into four 3*3 first sub-images based on the pixels of the first original image, that is, Figure 4a The first sub-image 1, the first sub-image 2, the first sub-image 3 and the first sub-image 4. The pixel points in the first sub-image 1 are 0-8, the pixel points in the first sub-image 2 are 10-18, the pixel points in the first sub-image 3 are 20-28, and the pixel points in the first sub-image 4 are 30-38.

[0083] S303 , selecting a preset number of pixels from each first sub-image to obtain at least one group of pixels corresponding to the multiple first sub-images, and determining at least one new first image for this round of amplification based on the at least one group of pixels.

[0084] Optionally, the preset number may be at least one of the following: 1, 2, 3, ...

[0085] For example, assuming that the first sub-image is a 3*3 image, the preset number may be at least one of the following: 1, 2, 3. That is, the computing device may select 1 pixel from each 3*3 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or, select 2 pixel points from each 3*3 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points; and / or, select 2 pixel points from each 3*3 first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points.

[0086] For example, assuming the preset number is 1, see Figure 4b , Figure 4b is a schematic diagram of a process for determining a new first image provided by an embodiment of the present application. Figure 4b As shown, the computing device may select one pixel from each of the four first sub-images to obtain at least one group of pixels corresponding to the four first sub-images (each group of pixels has four pixels), and determine a new first image based on the at least one group of pixels. For example, assuming that the computing device selects pixel 0 from first sub-image 1, pixel 10 from first sub-image 2, pixel 20 from first sub-image 3, and pixel 30 from first sub-image 4, the computing device may determine the new first image based on the four selected pixels. Figure 4bFor another example, assuming that the computing device selects pixel 1 from the first sub-image 1, pixel 11 from the first sub-image 2, pixel 21 from the first sub-image 3, and pixel 31 from the first sub-image 4, the computing device may determine based on the selected 4 pixels: Figure 4b For another example, assuming that the computing device selects 2 pixels from the first sub-image 1, 12 pixels from the first sub-image 2, 22 pixels from the first sub-image 3, and 32 pixels from the first sub-image 4, the computing device may determine based on the selected 4 pixels: Figure 4b Image 3 in.

[0087] For example, assuming the preset number is 2, see Figure 4c , Figure 4c FIG. 1 is another schematic diagram of a process for determining a new first image provided by an embodiment of the present application. Figure 4c As shown, the computing device can select 2 pixels from each of the 4 first sub-images to obtain at least one group of pixels corresponding to the 4 first sub-images (each group of pixels has 8 pixels), and determine a new first image based on at least one group of pixels. For example, the computing device can select 2 pixels in the horizontal direction from each of the 4 first sub-images, such as 0 and 1 pixels from the first sub-image 1; 10 and 11 pixels from the first sub-image 2; 20 and 21 pixels from the first sub-image 3; and 30 and 31 pixels from the first sub-image 4. In this case, the computing device can determine the new first image based on the 4 selected pixels. Figure 4c For another example, the computing device may select two pixels in the vertical direction from each of the four first sub-images, such as 0 and 3 pixels from the first sub-image 1; 10 and 13 pixels from the first sub-image 2; 20 and 23 pixels from the first sub-image 3; and 30 and 33 pixels from the first sub-image 4. In this case, the computing device may determine based on the four selected pixels: Figure 4c For another example, the computing device may select two pixels in the diagonal and anti-diagonal directions from the four first sub-images, such as 0 and 4 pixels selected from the first sub-image 1; 10 and 14 pixels selected from the first sub-image 2; 20 and 24 pixels selected from the first sub-image 3; and 30 and 34 pixels selected from the first sub-image 4. In this case, the computing device may determine based on the four selected pixels: Figure 4c Image 3 in.

[0088] For example, assuming the preset number is 3, see Figure 4d , Figure 4d FIG. 1 is another schematic diagram of a process for determining a new first image provided by an embodiment of the present application. Figure 4d As shown, the computing device can select 3 pixels from each of the 4 first sub-images to obtain at least one group of pixels corresponding to the 4 first sub-images (each group of pixels has 12 pixels), and determine a new first image based on at least one group of pixels. For example, the computing device can select 3 pixels in the horizontal direction from each of the 4 first sub-images, such as 0, 1, and 2 pixels from the first sub-image 1; 10, 11, and 12 pixels from the first sub-image 2; 20, 21, and 22 pixels from the first sub-image 3; and 30, 31, and 32 pixels from the first sub-image 4. In this case, the computing device can determine the new first image based on the 4 selected pixels. Figure 4d For another example, the computing device may select two pixels in the vertical direction from each of the four first sub-images, such as 0, 3, and 6 pixels from the first sub-image 1; 10, 13, and 16 pixels from the first sub-image 2; 20, 23, and 26 pixels from the first sub-image 3; and 30, 33, and 36 pixels from the first sub-image 4. In this case, the computing device may determine based on the four selected pixels: Figure 4d For another example, the computing device may select two pixels in the diagonal and anti-diagonal directions from the four first sub-images, such as 0, 4, and 8 pixels selected from the first sub-image 1; 10, 14, and 18 pixels selected from the first sub-image 2; 20, 24, and 28 pixels selected from the first sub-image 3; and 30, 34, and 38 pixels selected from the first sub-image 4. In this case, the computing device may determine based on the four selected pixels: Figure 4d Image 3 in.

[0089] S304. When it is determined that all combinations of a preset number of pixel points in each first sub-image have not been traversed, a step of selecting a preset number of pixel points from each first sub-image to obtain at least one group of pixel points corresponding to multiple first sub-images, and determining at least one new first image of a new round of amplification based on the at least one group of pixel points is performed, until all combinations of the preset number of pixel points in each first sub-image have been traversed, and a new first image is obtained by multiple rounds of amplification, thereby obtaining multiple new first images.

[0090] Exemplarily, assuming that the preset number is 1, if the computing device has not traversed all 9 pixels in each first sub-image, it is necessary to select the preset number of pixels from each first sub-image again to obtain at least one group of pixels corresponding to multiple first sub-images, and determine at least one new first image of a new round of amplification based on at least one group of pixels, until all 9 pixels in each first sub-image are traversed, and multiple new first images are obtained based on the new first images obtained by multiple rounds of amplification. In this case, the number of new first images that the computing device can obtain is

[0091] Similarly, when the preset number is 2, the number of new first images that the computing device can obtain is Similarly, when the preset number is 3, the number of new first images that the computing device can obtain is

[0092] S305 , selecting a preset number of pixels from each second sub-image to obtain at least one group of pixels corresponding to the plurality of second sub-images, and determining at least one new second image for this round of amplification based on the at least one group of pixels.

[0093] S306. When it is determined that all combinations of a preset number of pixel points in each second sub-image have not been traversed, execute the steps of selecting a preset number of pixel points from each second sub-image respectively to obtain at least one group of pixel points corresponding to the multiple second sub-images, and determine at least one new second image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new second images are obtained.

[0094] In an optional implementation, the relevant descriptions of steps S305 and S306 may refer to the descriptions of the aforementioned steps S303 and S304, respectively, and will not be repeated here.

[0095] S307. Construct an original sample pair based on the first original image and the second original image, and construct multiple augmented sample pairs based on multiple new first images and multiple new second images; the original sample pair and the multiple augmented sample pairs are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0096] In an embodiment of the present application, the computing device can obtain a new first image and a new second image by respectively amplifying the first original image and the second original image, and construct an original sample pair based on the first original image and the second original image, and construct an amplified sample pair based on the new first image and the new second image. In this way, when the original sample pair and the amplified sample pair are used to train the twin network to be trained, the twin network can learn more features based on the smallest original image (two original images, namely the first original image and the second original image), thereby improving the generalization ability of the twin network obtained through training.

[0097] In addition, compared with the method of not using the original sample pairs and amplified sample pairs obtained by the image data processing method to train the twin network to be trained, the use of this method can reduce the number of first original images obtained when the accuracy of the trained twin network is the same.

[0098] In an optional embodiment, Figure 2 and Figure 3 In the method shown, after the computing device constructs an original sample pair based on the first original image and the second original image, and constructs an amplified sample pair based on the new first image and the new second image, it can also train the twin network to be trained based on the original sample pair and the amplified sample pair to obtain a trained twin network.

[0099] In this embodiment, the structure of the twin network can be as follows Figure 5 As shown, Figure 5 Schematic diagram of a twin network structure provided in the embodiment of the present application. Figure 5 As shown, the twin network to be trained may include a first feature extraction network ( Figure 5 The first feature extraction network is a convolutional neural network 1, and the second feature extraction network ( Figure 5 In the figure, the second feature extraction network is a convolutional neural network 2), the first classification network and the second classification network. Optionally, the computing device trains the training twin network based on the original sample pair and the amplified sample pair, which may include but is not limited to the following steps:

[0100] Step 1: Determine a set of sample image pairs based on the original sample pairs and the amplified sample pairs.

[0101] Step 2: For each sample image pair in the sample image pair set, the first sample image in the sample image pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature corresponding to the first sample image, and the second sample image in the sample image pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature corresponding to the second sample image.

[0102] Step three, determine the total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained.

[0103] In an optional implementation, the computing device may use the following formula (1) to determine the total loss value of the sample image pair.

[0104] L = λ×(L cls (G w (D1′)+L cls (G w (D2′)))+(1-2λ)×L rps (1)

[0105] In formula (1), L represents the total loss value of the sample image pair; λ represents the loss weight; G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; L rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; L cls (G w (D1′) represents the second loss value, that is, the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object; L cls (G w (D2′) represents the third loss value, that is, the loss value between the second predicted behavior and the actual behavior of the sample object.

[0106] In an optional embodiment, the first loss value (ie, L rps ) can be determined by the computing device in the following manner: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0107] In this implementation, the sample image feature similarity between the first sample image feature and the second sample image feature can be determined by the computing device through a difference vector similarity measurement method, or it can be determined by the computing device through a measurement method combining distance similarity and cosine similarity, which is not limited here.

[0108] When the computing device determines the sample image feature similarity between the first sample image feature and the second sample image feature by measuring the difference vector similarity, the following formula (2) may be used.

[0109]

[0110] In formula (2), G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; e It means G w The number of elements in (D1′) that are not equal to 0; f It means G w The number of elements in (D2′) that are not equal to 0; ef It means G w (D1′) and G w The number of elements in the difference of (D2′) that are not equal to 0.

[0111] When the computing device determines the sample image feature similarity between the first sample image feature and the second sample image feature by a measurement method combining distance similarity and cosine similarity, the following formula (3) may be used.

[0112] S(G w (D1′), G w (D2′))=dis(G w (D1′), G w (D2′))·cos(G w (D1′), G w (D2′)) (3)

[0113] In formula (3), G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; dis() represents the distance similarity; and cos() represents the cosine similarity.

[0114] In this implementation, the preset value interval may be [0, 1]. Optionally, the computing device converts the sample image feature similarity into the sample image feature similarity probability within the preset value interval, which may be achieved by using Figure 5 The fully connected layer and sigmoid layer of the recognition network in the twin network to be trained shown in the figure convert the sample image feature similarity into a numerical value in the interval [0,1], namely, RPS, and use RPS as the sample image feature similarity probability.

[0115] It can be understood that if the probability of similarity of sample image features is higher, it means that the recognition result of the first classification network in the twin network for the sample object's to-be-identified behavior in the first sample image (full image) is more consistent with the recognition result of the second classification network for the sample object's to-be-identified behavior in the second sample image (partial image).

[0116] In this implementation, when the computing device determines the first loss value between the first sample image and the second sample image based on the similarity probability of the sample image features, the following formula (4) may be used.

[0117]

[0118] In formula (4), L rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; RPS represents the probability of similarity of sample image features; λ represents the loss weight; l represents the label of the sample pair, if the sample pair is a positive sample pair, l is 1, if the sample pair is a negative sample pair, l is 0; It represents the regularization function.

[0119] Step 4: When it is determined that the total loss value of any sample image pair does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the conditions for stopping training, thereby obtaining the trained twin network.

[0120] With this implementation, on the one hand, the computing device can determine a set of sample image pairs based on the original sample pairs and the augmented sample pairs; and train the twin network to be trained using the set of sample image pairs, so that the twin network can learn more features based on the smallest original image (two original images, i.e., the first original image and the second original image), thereby improving the generalization ability of the twin network after training. On the other hand, the computing device trains the twin network by using the total loss value of each sample image pair in the sample image set, wherein the total loss value of each sample image pair is composed of the first loss value between the first sample image and the second sample image in the sample image pair, the second loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the third loss value between the second predicted behavior and the actual behavior of the sample object; thereafter, when the total loss value of any sample image pair in the sample image pair set does not meet the stop training condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of the multiple sample image pairs meet the stop training condition. In this way, when the target object to be identified in the vehicle cabin is subsequently identified based on the trained twin network, the accuracy of the recognition result can be improved.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides an image data processing device for implementing the image data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image data processing device embodiments provided below can refer to the limitations of the image data processing method above, and will not be repeated here.

[0123] See also Figure 6 , Figure 6 Schematic diagram of the structure of an image data processing device provided in an embodiment of the present application. Figure 6 As shown, the image data processing device may include but is not limited to:

[0124] The acquisition and determination module 601 is used to acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0125] An image division module 602 is used to obtain a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image by dividing the pixels;

[0126] A determination module 603 is used to select a preset number of pixel points from each first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points;

[0127] The determination module 603 is further configured to select a preset number of pixel points from each second sub-image, obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determine a new second image based on the multiple groups of pixel points;

[0128] Construction module 604 is used to construct an original sample pair based on the first original image and the second original image, and to construct an augmented sample pair based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0129] In one embodiment, when the determination module 603 is used to select a preset number of pixel points from each first sub-image respectively, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points, it is specifically used to: select a preset number of pixel points from each first sub-image respectively, obtain at least one group of pixel points corresponding to multiple first sub-images, and determine at least one new first image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each first sub-image have not been traversed, execute the steps of selecting a preset number of pixel points from each first sub-image respectively, obtain at least one group of pixel points corresponding to multiple first sub-images, and determine at least one new first image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, and a new first image obtained by multiple rounds of amplification is obtained, a plurality of new first images are obtained.

[0130] In one embodiment, when the determination module 603 is used to select a preset number of pixel points from each second sub-image, obtain multiple groups of pixel points corresponding to multiple second sub-images, and determine a new second image based on the multiple groups of pixel points, it is specifically used to: select a preset number of pixel points from each second sub-image, obtain at least one group of pixel points corresponding to multiple second sub-images, and determine at least one new second image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each second sub-image have not been traversed, execute the steps of selecting a preset number of pixel points from each second sub-image, obtain at least one group of pixel points corresponding to multiple second sub-images, and determine at least one new second image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image are traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new second images are obtained.

[0131] In one embodiment, when the acquisition and determination module 601 is used to determine the second image based on the first image, it is specifically used to: input the first original image into a face detection model, and when it is detected that the first original image contains a sample object, output face information corresponding to the sample object; based on the face information, capture a face image corresponding to the sample object from the first original image; input the face image into a face key point detection model to obtain the face key point coordinates of the sample object; determine the target area based on the face key point coordinates, and obtain a second original image corresponding to the target area from the face image.

[0132] In one embodiment, when the image division module 602 is used to obtain multiple first sub-images corresponding to the first original image and multiple second sub-images corresponding to the second original image through pixel division, it is specifically used to: equally divide the pixel points of the first original image to obtain multiple first sub-images corresponding to the first original image after equal division; and equally divide the pixel points of the second original image to obtain multiple second sub-images corresponding to the second original image after equal division.

[0133] In one embodiment, the device also includes a training module; the training module is used to determine a set of sample image pairs based on the original sample pairs and the augmented sample pairs; for each sample image pair in the set of sample image pairs, the first sample image in the sample image pair is input into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and the second sample image in the sample image pair is input into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; determine the total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image feature The second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the condition for stopping training, thereby obtaining the trained twin network.

[0134] In one embodiment, the determination module 603 is also used to: determine the first loss value by: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining the first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0135] Each module in the above-mentioned image data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computing device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0136] In an exemplary embodiment, a computing device is provided, which can be applied in a vehicle. The computing device can be a terminal, and its internal structure diagram can be as follows: Figure 7As shown. The computing device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computing device is used to provide computing and control capabilities. The memory of the computing device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computing device is used to exchange information between the processor and an external device. The communication interface of the computing device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, the steps in an image data processing method are implemented. The display unit of the computing device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computing device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computing device, or an external keyboard, touchpad, etc.

[0137] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computing device or server to which the solution of the present application is applied. The specific computing device or server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] In an exemplary embodiment, the present application provides a computing device, including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, the following steps are implemented:

[0139] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0140] By pixel point division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained;

[0141] Selecting a preset number of pixel points from each first sub-image respectively to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determining a new first image based on the multiple groups of pixel points;

[0142] Selecting a preset number of pixel points from each second sub-image to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points;

[0143] An original sample pair is constructed based on the first original image and the second original image, and an amplified sample pair is constructed based on the new first image and the new second image; the original sample pair and the amplified sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0144] In one embodiment, when a processor executes a computer program to select a preset number of pixel points from each first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points, the following steps are specifically implemented: select a preset number of pixel points from each first sub-image, obtain at least one group of pixel points corresponding to multiple first sub-images, and determine at least one new first image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each first sub-image have not been traversed, execute the step of selecting a preset number of pixel points from each first sub-image, obtain at least one group of pixel points corresponding to multiple first sub-images, and determine at least one new first image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, and a new first image is obtained by multiple rounds of amplification, a plurality of new first images are obtained.

[0145] In one embodiment, when a processor executes a computer program to select a preset number of pixel points from each second sub-image, obtain multiple groups of pixel points corresponding to multiple second sub-images, and determine a new second image based on the multiple groups of pixel points, the following steps are specifically implemented: select a preset number of pixel points from each second sub-image, obtain at least one group of pixel points corresponding to multiple second sub-images, and determine at least one new second image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each second sub-image have not been traversed, execute the step of selecting a preset number of pixel points from each second sub-image, obtain at least one group of pixel points corresponding to multiple second sub-images, and determine at least one new second image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new second images are obtained.

[0146] In one embodiment, when a processor executes a computer program to determine a second original image based on a first original image, the following steps are specifically implemented: the first original image is input into a face detection model, and when it is detected that the first original image contains a sample object, face information corresponding to the sample object is output; based on the face information, a face image corresponding to the sample object is captured from the first original image; the face image is input into a face key point detection model to obtain the face key point coordinates of the sample object; a target area is determined based on the face key point coordinates, and a second original image corresponding to the target area is obtained from the face image.

[0147] In one embodiment, when a processor executes a computer program to obtain multiple first sub-images corresponding to a first original image and multiple second sub-images corresponding to a second original image through pixel point division, the following steps are specifically implemented: equally dividing the pixel points of the first original image to obtain multiple first sub-images corresponding to the first original image after being equally divided; and equally dividing the pixel points of the second original image to obtain multiple second sub-images corresponding to the second original image after being equally divided.

[0148] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the original sample pairs and the amplified sample pairs, a set of sample image pairs is determined; for each sample image pair in the set of sample image pairs, the first sample image in the sample image pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature corresponding to the first sample image, and the second sample image in the sample image pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature corresponding to the second sample image; the total loss value of the sample image pair is determined; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image feature The second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the condition for stopping training, thereby obtaining the trained twin network.

[0149] In one embodiment, the processor executing the computer program further implements the following steps: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0150] In an exemplary embodiment, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0151] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0152] By pixel point division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained;

[0153] Selecting a preset number of pixel points from each first sub-image respectively to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determining a new first image based on the multiple groups of pixel points;

[0154] Selecting a preset number of pixel points from each second sub-image to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points;

[0155] An original sample pair is constructed based on the first original image and the second original image, and an amplified sample pair is constructed based on the new first image and the new second image; the original sample pair and the amplified sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0156] In one embodiment, when a computer program is executed by a processor to select a preset number of pixel points from each first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points, the following steps are specifically implemented: select a preset number of pixel points from each first sub-image, obtain at least one group of pixel points corresponding to multiple first sub-images, and determine at least one new first image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each first sub-image have not been traversed, execute the step of selecting a preset number of pixel points from each first sub-image, obtain at least one group of pixel points corresponding to multiple first sub-images, and determine at least one new first image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new first images are obtained.

[0157] In one embodiment, when a computer program is executed by a processor to select a preset number of pixel points from each second sub-image, obtain multiple groups of pixel points corresponding to multiple second sub-images, and determine a new second image based on the multiple groups of pixel points, the following steps are specifically implemented: select a preset number of pixel points from each second sub-image, obtain at least one group of pixel points corresponding to multiple second sub-images, and determine at least one new second image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each second sub-image have not been traversed, execute the step of selecting a preset number of pixel points from each second sub-image, obtain at least one group of pixel points corresponding to multiple second sub-images, and determine at least one new second image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new second images are obtained.

[0158] In one embodiment, when a computer program is executed by a processor to determine a second original image based on a first original image, the following steps are specifically implemented: the first original image is input into a face detection model, and when it is detected that the first original image contains a sample object, face information corresponding to the sample object is output; based on the face information, a face image corresponding to the sample object is captured from the first original image; the face image is input into a face key point detection model to obtain the face key point coordinates of the sample object; a target area is determined based on the face key point coordinates, and a second original image corresponding to the target area is obtained from the face image.

[0159] In one embodiment, when a computer program is executed by a processor to obtain multiple first sub-images corresponding to a first original image and multiple second sub-images corresponding to a second original image through pixel point division, the following steps are specifically implemented: equally dividing the pixel points of the first original image to obtain multiple first sub-images corresponding to the first original image after equal division; and equally dividing the pixel points of the second original image to obtain multiple second sub-images corresponding to the second original image after equal division.

[0160] In one embodiment, the computer program is executed by the processor to further implement the following steps: based on the original sample pairs and the augmented sample pairs, determine a set of sample image pairs; for each sample image pair in the set of sample image pairs, input the first sample image in the sample image pair into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and input the second sample image in the sample image pair into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; determine the total loss value of the sample image pair; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image feature The second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the condition for stopping training, thereby obtaining the trained twin network.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0162] In an exemplary embodiment, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0163] Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object;

[0164] By pixel point division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained;

[0165] Selecting a preset number of pixel points from each first sub-image respectively to obtain multiple groups of pixel points corresponding to the multiple first sub-images, and determining a new first image based on the multiple groups of pixel points;

[0166] Selecting a preset number of pixel points from each second sub-image to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points;

[0167] An original sample pair is constructed based on the first original image and the second original image, and an augmented sample pair is constructed based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

[0168] In one embodiment, when a computer program is executed by a processor to select a preset number of pixel points from each first sub-image, obtain multiple groups of pixel points corresponding to multiple first sub-images, and determine a new first image based on the multiple groups of pixel points, the following steps are specifically implemented: select a preset number of pixel points from each first sub-image, obtain at least one group of pixel points corresponding to multiple first sub-images, and determine at least one new first image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each first sub-image have not been traversed, execute the step of selecting a preset number of pixel points from each first sub-image, obtain at least one group of pixel points corresponding to multiple first sub-images, and determine at least one new first image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new first images are obtained.

[0169] In one embodiment, when a computer program is executed by a processor to select a preset number of pixel points from each second sub-image, obtain multiple groups of pixel points corresponding to multiple second sub-images, and determine a new second image based on the multiple groups of pixel points, the following steps are specifically implemented: select a preset number of pixel points from each second sub-image, obtain at least one group of pixel points corresponding to multiple second sub-images, and determine at least one new second image of this round of amplification based on the at least one group of pixel points; when it is determined that all combinations of the preset number of pixel points in each second sub-image have not been traversed, execute the step of selecting a preset number of pixel points from each second sub-image, obtain at least one group of pixel points corresponding to multiple second sub-images, and determine at least one new second image of a new round of amplification based on the at least one group of pixel points; until all combinations of the preset number of pixel points in each first sub-image have been traversed, a new first image obtained by multiple rounds of amplification is obtained, and multiple new second images are obtained.

[0170] In one embodiment, when a computer program is executed by a processor to determine a second original image based on a first original image, the following steps are specifically implemented: the first original image is input into a face detection model, and when it is detected that the first original image contains a sample object, face information corresponding to the sample object is output; based on the face information, a face image corresponding to the sample object is captured from the first original image; the face image is input into a face key point detection model to obtain the face key point coordinates of the sample object; a target area is determined based on the face key point coordinates, and a second original image corresponding to the target area is obtained from the face image.

[0171] In one embodiment, when a computer program is executed by a processor to obtain multiple first sub-images corresponding to a first original image and multiple second sub-images corresponding to a second original image through pixel point division, the following steps are specifically implemented: equally dividing the pixel points of the first original image to obtain multiple first sub-images corresponding to the first original image after equal division; and equally dividing the pixel points of the second original image to obtain multiple second sub-images corresponding to the second original image after equal division.

[0172] In one embodiment, the computer program is executed by the processor to further implement the following steps: based on the original sample pairs and the augmented sample pairs, determine a set of sample image pairs; for each sample image pair in the set of sample image pairs, input the first sample image in the sample image pair into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and input the second sample image in the sample image pair into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; determine the total loss value of the sample image pair; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is based on the first sample image feature The second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when it is determined that the total loss value of any sample image pair does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample image pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample image pairs meet the condition for stopping training, thereby obtaining the trained twin network.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0174] It should be noted that the data involved in this application (including but not limited to the first original image of the sample object, the second original image of the sample object, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations. In addition, the new first image, the new second image, the original sample pair and the amplified sample pair involved in this application are also images authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant images must comply with relevant regulations.

[0175] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0176] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0177] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing image data, characterized in that: The method comprises: Acquire a first original image of the sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; By pixel division, a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image are obtained; Selecting a preset number of pixel points from each of the first sub-images to obtain multiple groups of pixel points corresponding to the first sub-images, and determining a new first image based on the multiple groups of pixel points; Selecting the preset number of pixel points from each of the second sub-images respectively to obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determining a new second image based on the multiple groups of pixel points; An original sample pair is constructed based on the first original image and the second original image, and an augmented sample pair is constructed based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train the twin network to be trained, and the trained twin network is used to identify the target object in the vehicle cabin.

2. The method according to claim 1, characterized in that: The selecting a preset number of pixel points from each of the first sub-images to obtain a plurality of groups of pixel points corresponding to the plurality of the first sub-images, and determining a new first image based on the plurality of groups of pixel points includes: Selecting a preset number of pixel points from each of the first sub-images to obtain at least one group of pixel points corresponding to the plurality of the first sub-images, and determining at least one new first image of this round of amplification based on the at least one group of pixel points; When it is determined that all combinations of the preset number of pixel points in each of the first sub-images have not been traversed, performing the steps of selecting the preset number of pixel points from each of the first sub-images to obtain at least one group of pixel points corresponding to the plurality of first sub-images, and determining at least one new first image of a new round of enlargement based on the at least one group of pixel points; Until all combinations of a preset number of pixel points in each of the first sub-images are traversed, a new first image is obtained by multiple rounds of amplification to obtain multiple new first images.

3. The method according to claim 2, characterized in that The step of selecting the preset number of pixel points from each of the second sub-images to obtain a plurality of groups of pixel points corresponding to the plurality of second sub-images, and determining a new second image based on the plurality of groups of pixel points includes: Selecting a preset number of pixel points from each of the second sub-images to obtain at least one group of pixel points corresponding to the plurality of second sub-images, and determining at least one new second image of this round of amplification based on the at least one group of pixel points; When it is determined that all combinations of the preset number of pixel points in each of the second sub-images have not been traversed, performing the steps of selecting the preset number of pixel points from each of the second sub-images to obtain at least one group of pixel points corresponding to the plurality of second sub-images, and determining at least one new second image of a new round of enlargement based on the at least one group of pixel points; Until all combinations of a preset number of pixel points in each of the first sub-images are traversed, a new first image is obtained by multiple rounds of amplification to obtain multiple new second images.

4. The method according to claim 1, characterized in that The determining the second original image based on the first original image comprises: Inputting the first original image into a face detection model, and when detecting that the first original image contains the sample object, outputting face information corresponding to the sample object; Based on the facial information, extracting a facial image corresponding to the sample object from the first original image; Inputting the face image into a face key point detection model to obtain the coordinates of the face key points of the sample object; The target area is determined based on the coordinates of the facial key points, and a second original image corresponding to the target area is acquired from the facial image.

5. The method according to claim 1, characterized in that The step of obtaining a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image by pixel division includes: equally dividing the pixels of the first original image to obtain a plurality of equally divided first sub-images corresponding to the first original image; and The pixels of the second original image are equally divided to obtain a plurality of equally divided second sub-images corresponding to the second original image.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Determine a set of sample image pairs based on the original sample pairs and the amplified sample pairs; For each sample image pair in the sample image pair set, input the first sample image in the sample image pair into the first feature extraction network included in the twin network to be trained to obtain a first sample image feature corresponding to the first sample image, and input the second sample image in the sample image pair into the second feature extraction network included in the twin network to be trained to obtain a second sample image feature corresponding to the second sample image; Determine the total loss value of the sample image pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is a loss value between a first predicted behavior of the sample object and an actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into a first classification network included in the twin network to be trained; the third loss value is a loss value between a second predicted behavior of the sample object and the actual behavior, and the second predicted behavior is obtained by inputting the second sample image feature into a second classification network included in the twin network to be trained; When it is determined that the total loss value of any of the sample image pairs does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each of the sample image pairs, and the twin network to be trained is trained again until the total loss values ​​of the multiple sample image pairs respectively meet the condition for stopping training, thereby obtaining the trained twin network.

7. The method according to claim 6, characterized in that The first loss value is determined in the following manner: determining a sample image feature similarity between the first sample image feature and the second sample image feature; Converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; Based on the sample image feature similarity probability, a first loss value between the first sample image and the second sample image is determined.

8. An image data processing device, characterized in that: The device comprises: an acquisition and determination module, configured to acquire a first original image of a sample object, and determine a second original image based on the first original image; the second original image is a local image corresponding to a target area in the first original image, and the target area is an area related to the behavior to be identified of the sample object; An image division module, configured to obtain a plurality of first sub-images corresponding to the first original image and a plurality of second sub-images corresponding to the second original image by dividing the pixels; a determination module, configured to select a preset number of pixel points from each of the first sub-images, obtain a plurality of groups of pixel points corresponding to the plurality of the first sub-images, and determine a new first image based on the plurality of groups of pixel points; The determination module is further configured to select the preset number of pixel points from each of the second sub-images, obtain multiple groups of pixel points corresponding to the multiple second sub-images, and determine a new second image based on the multiple groups of pixel points; A construction module is used to construct an original sample pair based on the first original image and the second original image, and to construct an augmented sample pair based on the new first image and the new second image; the original sample pair and the augmented sample pair are used together to train a twin network to be trained, and the trained twin network is used to identify the behavior to be identified of the target object in the vehicle cabin.

9. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images