Image processing method and device, neural network training method and device

Abstract

The present disclosure relates to an image processing method and device, a neural network training method and device, the method comprising: performing target detection on a first image to obtain the center point coordinate information of a target object in the first image and the size information of the target object; determining the truncation point coordinates according to the center point coordinate information of the target object and the size information of the target object; determining the truncation area in the first image according to the truncation point coordinates and the truncation direction; truncation processing is performed on the truncation area in the first image to obtain a second image. The embodiments of the present disclosure can improve the training accuracy of the neural network.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method and device, and a neural network training method and device. Background Art

[0002] Object pose estimation refers to the process of estimating the pose information of the target object relative to the camera from the image, which can include: rotation (3 degrees of freedom) and translation (3 degrees of freedom). Rotation refers to the rotation relationship between the camera coordinate system and the target object coordinate system, and translation refers to the translation information of the origin of the camera coordinate system relative to the origin of the target object coordinate system. Object pose estimation plays an important role in many fields, such as "robotic operations", "autonomous driving", "augmented reality" and other fields.

[0003] In related technologies, object posture estimation algorithms are easily affected by external environmental factors such as noise, blur, occlusion, lighting changes, and cluttered background, which poses great challenges to accurately estimating the object's posture. Summary of the invention

[0004] The present disclosure proposes an image processing technology solution for intercepting an image.

[0005] According to one aspect of the present disclosure, there is provided an image processing method, comprising:

[0006] Performing target detection on the first image to obtain center point coordinate information of a target object in the first image and size information of the target object;

[0007] Determining the coordinates of the cutoff point according to the center point coordinate information of the target object and the size information of the target object;

[0008] Determining a truncation area in the first image according to the truncation point coordinates and the truncation direction;

[0009] The truncated area in the first image is truncated to obtain a second image.

[0010] In a possible implementation, determining the coordinates of the truncation point according to the center point coordinate information of the target object and the size information of the target object includes:

[0011] Determine a truncation point coordinate value range according to a preset truncation range coefficient, the center point coordinate information of the target object, and the size information of the target object;

[0012] The coordinates of the truncation point are obtained by selecting values ​​within the truncation point coordinate value range.

[0013] In a possible implementation manner, determining the truncation area in the first image according to the truncation point coordinates and the truncation direction includes:

[0014] Determine a first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point, and the truncation direction;

[0015] All the first pixel points constitute an initial truncation area;

[0016] A truncation region in the first image is determined according to the initial truncation region.

[0017] In a possible implementation manner, determining the first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point, and the truncation direction includes:

[0018] For any pixel point, if the truncation direction is upward and the ordinate of the coordinates of the truncation point is smaller than the ordinate of the pixel point, the pixel point is determined to be the first pixel point; or,

[0019] If the truncation direction is downward, and the ordinate of the truncation point coordinate is greater than the ordinate of the pixel point, then the pixel point is determined to be the first pixel point; or,

[0020] If the truncation direction is to the left, and the horizontal coordinate of the truncation point coordinate is greater than the horizontal coordinate of the pixel point, then the pixel point is determined to be the first pixel point; or,

[0021] If the truncation direction is rightward, and the horizontal coordinate of the truncation point coordinate is smaller than the horizontal coordinate of the pixel point, the pixel point is determined to be the first pixel point.

[0022] In a possible implementation manner, determining the truncation area in the first image according to the initial truncation area includes:

[0023] A truncation area in the first image is determined according to the center point coordinate information of the target object, the size information of the target object and the initial truncation area.

[0024] In a possible implementation manner, the truncating the truncated area in the first image to obtain the second image includes:

[0025] The pixel values ​​corresponding to each pixel point in the truncated area are transformed to obtain the second image.

[0026] In a possible implementation manner, the truncating the truncated area in the first image to obtain the second image includes:

[0027] Extracting an occlusion image from the third image, wherein the size of the occlusion image is the same as the size of the truncation area;

[0028] The truncated area in the first image is replaced with the occluded image to obtain the second image.

[0029] According to one aspect of the present disclosure, a neural network training method is provided, the method comprising:

[0030] The neural network is trained according to a preset training set, wherein the sample images in the training set are obtained after truncation processing using any of the aforementioned image processing methods.

[0031] According to one aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0032] A detection module, used to perform target detection on the first image to obtain the center point coordinate information of the target object in the first image and the size information of the target object;

[0033] A first determining module, used to determine the coordinates of the truncation point according to the center point coordinate information of the target object and the size information of the target object;

[0034] A second determination module, configured to determine a truncation area in the first image according to the truncation point coordinates and the truncation direction;

[0035] A processing module is used to perform truncation processing on the truncation area in the first image to obtain a second image.

[0036] In a possible implementation manner, the first determining module is further configured to:

[0037] Determine a truncation point coordinate value range according to a preset truncation range coefficient, the center point coordinate information of the target object, and the size information of the target object;

[0038] The coordinates of the truncation point are obtained by selecting values ​​within the truncation point coordinate value range.

[0039] In a possible implementation manner, the second determining module is further configured to:

[0040] Determine a first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point, and the truncation direction;

[0041] All the first pixel points constitute an initial truncation area;

[0042] A truncation region in the first image is determined according to the initial truncation region.

[0043] In a possible implementation manner, the second determining module is further configured to:

[0044] For any pixel point, if the truncation direction is upward and the ordinate of the coordinates of the truncation point is smaller than the ordinate of the pixel point, the pixel point is determined to be the first pixel point; or,

[0045] If the truncation direction is downward, and the ordinate of the truncation point coordinate is greater than the ordinate of the pixel point, then the pixel point is determined to be the first pixel point; or,

[0046] If the truncation direction is to the left, and the horizontal coordinate of the truncation point coordinate is greater than the horizontal coordinate of the pixel point, then the pixel point is determined to be the first pixel point; or,

[0047] If the truncation direction is rightward, and the horizontal coordinate of the truncation point coordinate is smaller than the horizontal coordinate of the pixel point, the pixel point is determined to be the first pixel point.

[0048] In a possible implementation manner, the second determining module is further configured to:

[0049] A truncation area in the first image is determined according to the center point coordinate information of the target object, the size information of the target object and the initial truncation area.

[0050] In a possible implementation, the processing module is further configured to:

[0051] The pixel values ​​corresponding to each pixel point in the truncated area are transformed to obtain the second image.

[0052] In a possible implementation, the processing module is further configured to:

[0053] Extracting an occlusion image from the third image, wherein the size of the occlusion image is the same as the size of the truncation area;

[0054] The truncated area in the first image is replaced with the occluded image to obtain the second image.

[0055] According to one aspect of the present disclosure, a neural network training device is provided, the device comprising:

[0056] The training module is used to train the neural network according to a preset training set, wherein the sample images in the training set are obtained after truncation processing using any of the aforementioned image processing methods.

[0057] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0058] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0059] In this way, target detection can be performed on the first image to obtain the center point coordinate information of the target object in the first image and the size information of the target object, and the truncation point coordinates can be determined based on the center point coordinate information of the target object and the size information of the target object. According to the truncation point coordinates and the truncation direction, the truncation area in the first image is determined, and the truncation area in the first image is truncate to obtain a second image. According to the image processing method and device, neural network training method and device provided by the embodiments of the present disclosure, a second image that can simulate a situation where an object is severely occluded can be obtained, and the second image can be used to train a neural network for object posture estimation, thereby improving the robustness and accuracy of the trained neural network to occlusion.

[0060] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0062] Figure 1 A flowchart showing an image processing method according to an embodiment of the present disclosure;

[0063] Figure 2 A schematic diagram showing an image processing method according to an embodiment of the present disclosure;

[0064] Figure 3(a) to Figure 3(c) A schematic diagram showing an image processing method according to an embodiment of the present disclosure;

[0065] Figure 4 A block diagram showing an image processing apparatus according to an embodiment of the present disclosure;

[0066] Figure 5 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown;

[0067] Figure 6 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0068] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0069] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0070] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0071] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0072] In the related art, the sample image is partially occluded by randomly introducing occlusion blocks in the sample image, so as to improve the robustness of the neural network to occlusion. However, the improvement of the robustness of occlusion by the random occlusion method is often very limited, because when the occlusion block is too small, it can only occlude a small part of the target object, and it is impossible to simulate the situation of severe occlusion. The trained neural network is not very robust to occlusion; but if the occlusion block is too large, it is easy to completely cover the target object, resulting in a sample image that does not include the target object, which affects the training.

[0073] The disclosed embodiment provides an image processing method, which can occlude a target image in a first image, and obtain a second image that simulates a severe occlusion situation without completely occluding the target object.

[0074] Figure 1A flowchart of an image processing method according to an embodiment of the present disclosure is shown. In a possible implementation, the image processing method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method can be executed by a server.

[0075] like Figure 1 As shown, the image processing method includes:

[0076] In step S11, target detection is performed on the first image to obtain the center point coordinate information of the target object in the first image and the size information of the target object.

[0077] For example, the first image may be an image to be occluded by a target object, such as a face, a human body, an animal, a plant, or an object. Target detection may be performed on the first image, including detecting the target object and the coordinate information of the center point of the target object (c x , c y ), the size information of the target object (l, w), where l is the length of the target object and w is the width of the target object.

[0078] Exemplarily, a pre-trained neural network for detecting center point coordinate information and size information of a target object may be used to perform target detection on the first image to obtain center point coordinate information and size information of the target object in the first image.

[0079] In step S12, the coordinates of the truncation point are determined according to the center point coordinate information of the target object and the size information of the target object.

[0080] For example, the above-mentioned truncation point coordinates are coordinate information used to determine the truncation area. According to the center point coordinate information of the target object and the size information of the target object, a truncation point coordinate can be determined, and the truncation point coordinate is located on an edge of the truncation area.

[0081] In a possible implementation, determining the coordinates of the truncation point according to the center point coordinate information of the target object and the size information of the target object may include:

[0082] Determine a truncation point coordinate value range according to a preset truncation range coefficient, the center point coordinate information of the target object, and the size information of the target object;

[0083] The coordinates of the truncation point are obtained by selecting values ​​within the truncation point coordinate value range.

[0084] For example, a truncation range coefficient can be preset, such as setting the preset truncation range coefficient to 0.1. The floating value of the horizontal coordinate of the truncation point can be determined based on the preset truncation range coefficient and the length of the target object (for example, the floating value of the horizontal coordinate of the truncation point can be the product of the truncation range coefficient and the length of the target object). The value range of the horizontal coordinate in the truncation point coordinates is determined based on the horizontal coordinate in the center point coordinate information of the target object and the floating value of the horizontal coordinate of the truncation point. Similarly, the floating value of the vertical coordinate of the truncation point can be determined based on the preset truncation range coefficient and the width of the target object (for example, the floating value of the vertical coordinate of the truncation point can be the product of the truncation range coefficient and the width of the target object). The value range of the vertical coordinate in the truncation point coordinates is determined based on the vertical coordinate in the center point coordinate information of the target object and the floating value of the vertical coordinate of the truncation point. In this way, the value range of the truncation point coordinates can be obtained.

[0085] In step S13, a truncation area in the first image is determined according to the truncation point coordinates and the truncation direction.

[0086] For example, multiple optional truncation directions may be preset, and one optional truncation direction may be randomly selected from the multiple optional truncation directions as the truncation direction. For example, four optional truncation directions may be preset: upward; down left To the right

[0087] It should be noted that the above-mentioned setting of four optional truncation directions is an example of an embodiment of the present disclosure, and is not a limitation on the optional truncation directions in the embodiment of the present disclosure. In fact, any direction can be set as an optional truncation direction, for example: upper left, lower left, upper right, lower right, etc.

[0088] Exemplarily, the coordinates of the cutoff point and the cutoff direction may be determined according to formula (1).

[0089]

[0090] Wherein, η is a preset cutoff range coefficient, and U is the probability distribution followed by the sampling. For example, U can be set to a uniform distribution. represents n different optional truncation directions for truncating the first image, is the coordinate of the cutoff point, is the cutoff direction. From formula (1), we can know that The value range is [c x -η·l,c x +η·l], The value range is [c y -η·w,c y +η·w].

[0091] It should be noted that the above-mentioned randomly selected truncation direction is an example of determining the truncation direction in the embodiment of the present disclosure. In fact, the truncation direction can also be determined based on the relationship between the truncation point coordinates and the center point coordinate information of the target object. For example, when the truncation point coordinates are above the center point of the target object, the truncation direction can be determined to be upward; when the truncation point coordinates are below the center point of the target object, the truncation direction can be determined to be downward.

[0092] After determining the truncation point coordinates and the truncation direction, the truncation area in the first image can be determined. For example, in the first image, the side indicated by the truncation direction of the truncation point coordinates is determined as the truncation area. For example, if the truncation direction is left, the image area on the left side of the truncation point coordinates in the first image can be determined as the truncation area.

[0093] In step S14, the truncated area in the first image is truncated to obtain a second image.

[0094] For example, after determining the truncated area in the first image, the truncated area can be truncated to obtain the second image. Exemplarily, the second image can be used to train a neural network, such as a neural network for object pose estimation, to improve the robustness of the neural network to occlusion.

[0095] In this way, target detection can be performed on the first image to obtain the center point coordinate information of the target object in the first image and the size information of the target object, and the truncation point coordinates can be determined based on the center point coordinate information of the target object and the size information of the target object. According to the truncation point coordinates and the truncation direction, the truncation area in the first image is determined, and the truncation area in the first image is truncate to obtain a second image. According to the image processing method provided by the embodiment of the present disclosure, a second image that can simulate a situation where an object is severely occluded can be obtained, and the second image can be used to train a neural network for object posture estimation, thereby improving the robustness and accuracy of the trained neural network to occlusion.

[0096] In a possible implementation manner, determining the truncation area in the first image according to the truncation point coordinates and the truncation direction may include:

[0097] Determine a first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point, and the truncation direction;

[0098] All the first pixel points constitute an initial truncation area;

[0099] A truncation region in the first image is determined according to the initial truncation region.

[0100] For example, after determining the truncation point coordinates and the truncation direction, it is possible to determine whether any pixel in the first image is a first pixel based on the coordinates of the truncation point coordinates and the truncation direction. The first pixel is a pixel located in the initial truncation area. Exemplarily, the first pixel can be determined by the following formula (II).

[0101]

[0102] Among them, (x, y) is the coordinate of the pixel point, is the vector whose coordinates point to the pixel (x, y), that is For vector With cutoff direction The inner product of Sgn is the sign function. For zero or integer, sgn outputs 1, and for negative numbers, sgn outputs 0. t When (x, y) = 0, the pixel does not belong to the initial truncation area. t When (x, y)=1, the pixel point belongs to the initial truncation area, and the pixel point is the first pixel point.

[0103] In a possible implementation manner, determining the first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point, and the truncation direction may include:

[0104] For any pixel point, if the truncation direction is upward and the ordinate of the coordinates of the truncation point is smaller than the ordinate of the pixel point, the pixel point is determined to be the first pixel point; or,

[0105] If the truncation direction is downward, and the ordinate of the truncation point coordinate is greater than the ordinate of the pixel point, then the pixel point is determined to be the first pixel point; or,

[0106] If the truncation direction is to the left, and the horizontal coordinate of the truncation point coordinate is greater than the horizontal coordinate of the pixel point, then the pixel point is determined to be the first pixel point; or,

[0107] If the truncation direction is rightward, and the horizontal coordinate of the truncation point coordinate is smaller than the horizontal coordinate of the pixel point, the pixel point is determined to be the first pixel point.

[0108] For example, after the truncation point coordinates and the truncation direction are determined, the first pixel point may be determined according to the truncation point coordinates and the truncation direction.

[0109] For example, if Figure 2In the coordinate system shown, for any pixel point, if the truncation direction is upward, when the ordinate of the pixel point is greater than the ordinate in the truncation point coordinates, it can be determined that the pixel point is above the truncation point coordinates, so the pixel point can be determined as the first pixel point; if the truncation direction is downward, when the ordinate of the pixel point is less than the ordinate in the truncation point coordinates, it can be determined that the pixel point is below the truncation point coordinates, so the pixel point can be determined as the first pixel point; if the truncation direction is to the left, when the abscissa of the pixel point is less than the abscissa in the truncation point coordinates, it can be determined that the pixel point is to the left of the truncation point coordinates, so the pixel point can be determined as the first pixel point; if the truncation direction is to the right, when the abscissa of the pixel point is greater than the abscissa in the truncation point coordinates, it can be determined that the pixel point is to the right of the truncation point coordinates, so the pixel point can be determined as the first pixel point.

[0110] After determining the first pixel point, all first pixel points can form an initial truncation area, and then based on the initial truncation area, the truncation area in the first image can be determined. For example, the initial truncation area can be used as the truncation area in the first image, or a partial area in the initial truncation area can be determined as the truncation area in the first image.

[0111] In a possible implementation manner, determining the truncation area in the first image according to the initial truncation area may include:

[0112] A truncation area in the first image is determined according to the center point coordinate information of the target object, the size information of the target object and the initial truncation area.

[0113] For example, after the initial truncation region in the first image is determined, the truncation region may be determined from the initial truncation region according to the coordinate information of the center point of the target object and the size information of the target object.

[0114] For example, when the truncation direction is upward, it can be determined that the ordinate in the initial truncation area is located at With w / 2+c y The pixel points between are the pixel points in the truncated area, so as to obtain the truncated area, such as Figure 2 As shown; when the truncation direction is downward, it can be determined that the ordinate in the initial truncation area is located at and The pixel points between are the pixel points in the truncation area, so as to obtain the truncation area; when the truncation direction is to the left, it can be determined that the horizontal coordinate in the initial truncation area is located at With c x The pixel points between -l / 2 are the pixel points in the truncation area, so as to obtain the truncation area; when the truncation direction is to the right, it can be determined that the horizontal coordinate in the initial truncation area is located at With c xThe pixel points between +l / 2 are the pixel points in the truncation area, thereby obtaining the truncation area.

[0115] In fact, if the object area of ​​the target object is known (for example, the edge contour information of the target object has been obtained), then according to the selected truncation point coordinates and truncation direction, only the object area corresponding to the target object in the initial truncation area can be truncated, and corresponding pixel conversion and other processing can be performed, while the background area that does not belong to the target object will not be changed.

[0116] In a possible implementation manner, the performing truncation processing on the truncation area in the first image to obtain the second image may include:

[0117] The pixel values ​​corresponding to each pixel point in the truncated area are transformed to obtain the second image.

[0118] For example, the pixel value corresponding to each pixel point in the truncated area can be set to a value such as 0 or 1, or the pixel value corresponding to each pixel point in the truncated area can be randomly set to an arbitrary value to transform, mask, etc. the original image content in the truncated area to obtain a second image, refer to Figure 3(a) and Figure 3(b).

[0119] In a possible implementation manner, performing occlusion processing on the truncated area in the first image to obtain the second image may include:

[0120] Extracting an occlusion image from the third image, wherein the size of the occlusion image is the same as the size of the truncation area;

[0121] The truncated area in the first image is replaced with the occluded image to obtain the second image.

[0122] Among them, the third image can be any image different from the first image. An occluded image with the same size as the truncated area can be cut out in the third image, and the truncated area in the first image can be replaced with the occluded image. The original image content in the truncated area can be covered and processed to obtain the second image, refer to Figure 3(a) and Figure 3(c).

[0123] The present disclosure provides a neural network training method, which may include:

[0124] The neural network is trained according to a preset training set, wherein the sample images in the training set are obtained after truncation processing using any of the aforementioned image processing methods.

[0125] For example, the neural network can be used for object posture estimation, and the neural network can be used in the fields of "robotic operation", "autonomous driving", "augmented reality", etc. Any of the above-mentioned image processing methods can be used to truncate the image to obtain a sample image that simulates a severe occlusion without completely occluding the target object. Since the truncation operation does not change the posture information of the object in the image, the sample images obtained after truncation are used to replace the original sample images to train the neural network.

[0126] In this way, according to the neural network training method provided by the embodiment of the present disclosure, the neural network can be trained by using a sample image obtained after truncation processing using any of the aforementioned image processing methods, thereby improving the robustness of the trained neural network to occlusion.

[0127] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0128] In addition, the present disclosure also provides an image processing device, a neural network training device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any image processing method and neural network training method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0129] Figure 4 A block diagram of an image processing device according to an embodiment of the present disclosure is shown. Figure 4 As shown, the device comprises:

[0130] The detection module 41 may be used to perform target detection on the first image to obtain the center point coordinate information of the target object in the first image and the size information of the target object;

[0131] The first determination module 42 may be used to determine the coordinates of the truncation point according to the center point coordinate information of the target object and the size information of the target object;

[0132] A second determination module 43 may be used to determine a truncation area in the first image according to the truncation point coordinates and the truncation direction;

[0133] The processing module 44 may be configured to perform truncation processing on the truncation area in the first image to obtain a second image.

[0134] In this way, target detection can be performed on the first image to obtain the center point coordinate information of the target object in the first image and the size information of the target object, and the truncation point coordinates can be determined based on the center point coordinate information of the target object and the size information of the target object. According to the truncation point coordinates and the truncation direction, the truncation area in the first image is determined, and the truncation area in the first image is truncate to obtain a second image. According to the image processing device provided by the embodiment of the present disclosure, a second image that can simulate a situation where an object is severely occluded can be obtained, and the second image can be used to train a neural network for object posture estimation, thereby improving the robustness and accuracy of the trained neural network to occlusion.

[0135] In a possible implementation, the first determining module 42 may also be configured to:

[0136] Determine a truncation point coordinate value range according to a preset truncation range coefficient, the center point coordinate information of the target object, and the size information of the target object;

[0137] The coordinates of the truncation point are obtained by selecting values ​​within the truncation point coordinate value range.

[0138] In a possible implementation, the second determining module 43 may also be used to:

[0139] Determine a first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point, and the truncation direction;

[0140] All the first pixel points constitute an initial truncation area;

[0141] A truncation region in the first image is determined according to the initial truncation region.

[0142] In a possible implementation manner, the second determining module is further configured to:

[0143] For any pixel point, if the truncation direction is upward and the ordinate of the coordinates of the truncation point is smaller than the ordinate of the pixel point, the pixel point is determined to be the first pixel point; or,

[0144] If the truncation direction is downward, and the ordinate of the truncation point coordinate is greater than the ordinate of the pixel point, then the pixel point is determined to be the first pixel point; or,

[0145] If the truncation direction is to the left, and the horizontal coordinate of the truncation point coordinate is greater than the horizontal coordinate of the pixel point, then the pixel point is determined to be the first pixel point; or,

[0146] If the truncation direction is rightward, and the horizontal coordinate of the truncation point coordinate is smaller than the horizontal coordinate of the pixel point, the pixel point is determined to be the first pixel point.

[0147] In a possible implementation, the second determining module 43 may also be used to:

[0148] A truncation area in the first image is determined according to the center point coordinate information of the target object, the size information of the target object and the initial truncation area.

[0149] In a possible implementation, the processing module 44 may also be used to:

[0150] The pixel values ​​corresponding to each pixel point in the truncated area are transformed to obtain the second image.

[0151] In a possible implementation, the processing module 44 may also be used to:

[0152] Extracting an occlusion image from the third image, wherein the size of the occlusion image is the same as the size of the truncation area;

[0153] The truncated area in the first image is replaced with the occluded image to obtain the second image.

[0154] According to one aspect of the present disclosure, a neural network training device is provided, the device comprising:

[0155] The training module can be used to train a neural network according to a preset training set, wherein the sample images in the training set are obtained after truncation processing using any of the aforementioned image processing methods.

[0156] In this way, according to the neural network training device provided by the embodiment of the present disclosure, the neural network can be trained by using the sample image obtained after truncation processing using any of the aforementioned image processing methods, thereby improving the robustness of the trained neural network to occlusion.

[0157] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0158] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0159] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0160] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code. When the computer-readable code is executed on a device, a processor in the device executes instructions for implementing the image processing method or neural network training method provided in any of the above embodiments.

[0161] The embodiments of the present disclosure also provide another computer program product for storing computer-readable instructions, which, when executed, enable a computer to perform the operations of the image processing method or neural network training method provided in any of the above embodiments.

[0162] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0163] Figure 5 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0164] Reference Figure 5 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0165] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0166] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0167] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0168] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0169] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0170] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0171] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0172] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0173] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0174] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0175] Figure 61 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Figure 6 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0176] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2000). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0177] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0178] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0179] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0180] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0181] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0182] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0183] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0184] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0185] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0186] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0187] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, It is characterized in that include: Performing target detection on the first image to obtain center point coordinate information of a target object in the first image and size information of the target object; Determining the coordinates of the cutoff point according to the center point coordinate information of the target object and the size information of the target object; Determining a truncation area in the first image according to the truncation point coordinates and the truncation direction; performing truncation processing on the truncation area in the first image to obtain a second image; The determining the coordinates of the truncation point according to the center point coordinate information of the target object and the size information of the target object includes: Determine a truncation point coordinate value range according to a preset truncation range coefficient, the center point coordinate information of the target object, and the size information of the target object; Select a value within the truncation point coordinate value range to obtain the truncation point coordinate; The truncation point coordinates and the truncation direction are determined according to the following formula: Wherein, η is the preset cutoff range coefficient; U is the probability distribution followed by the sampling, specifically uniform distribution; represents n different optional truncation directions for truncating the first image; are the coordinates of the cutoff point; is the truncation direction; (c x ,c y ) is the coordinate of the center point of the target object; l is the length of the target object; w is the width of the target object; The value range is [c x -η·l,c x +η·l]; The value range is [c y -η·w,c y +η·w]; The determining the truncation area in the first image according to the truncation point coordinates and the truncation direction includes: Determine a first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point, and the truncation direction; All the first pixel points constitute an initial truncation area; Determining a truncation region in the first image according to the initial truncation region; The determining, according to the initial truncation area, a truncation area in the first image includes: Determine a truncation area in the first image according to the center point coordinate information of the target object, the size information of the target object and the initial truncation area; Wherein, when the truncation direction is upward, it is determined that the ordinate in the initial truncation area is located at With w / 2+c y The pixel points between are the pixel points in the truncated area, so as to obtain the truncated area; when the truncation direction is downward, it is determined that the ordinate in the initial truncation area is located at With w / 2-c y The pixel points between are the pixel points in the truncated area, so as to obtain the truncated area; when the truncation direction is to the left, it is determined that the horizontal coordinate in the initial truncation area is located at With c x -l / 2 are the pixel points in the truncated area, so as to obtain the truncated area; when the truncation direction is rightward, it is determined that the horizontal coordinate in the initial truncation area is located at With c x The pixel points between +l / 2 are the pixel points in the truncation area, thereby obtaining the truncation area.

2. The method according to claim 1, It is characterized in that The determining the first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point and the truncation direction includes: For any pixel point, if the truncation direction is upward and the ordinate of the coordinates of the truncation point is smaller than the ordinate of the pixel point, the pixel point is determined to be the first pixel point; or, If the truncation direction is downward, and the ordinate of the truncation point coordinate is greater than the ordinate of the pixel point, then the pixel point is determined to be the first pixel point; or, If the truncation direction is to the left, and the horizontal coordinate of the truncation point coordinate is greater than the horizontal coordinate of the pixel point, then the pixel point is determined to be the first pixel point; or, If the truncation direction is rightward, and the horizontal coordinate of the truncation point coordinate is smaller than the horizontal coordinate of the pixel point, the pixel point is determined to be the first pixel point.

3. The method according to claim 1 or 2, It is characterized in that The truncating the truncated area in the first image to obtain the second image includes: The pixel values ​​corresponding to each pixel point in the truncated area are transformed to obtain the second image.

4. The method according to claim 1 or 2, It is characterized in that The truncating the truncated area in the first image to obtain the second image includes: Extracting an occlusion image from the third image, wherein the size of the occlusion image is the same as the size of the truncation area; The truncated area in the first image is replaced with the occluded image to obtain the second image.

5. A neural network training method for object pose estimation, It is characterized in that The method comprises: A neural network is trained according to a preset training set, wherein the sample images in the training set are obtained after truncation processing using the image processing method according to any one of claims 1 to 4.

6. An image processing device, It is characterized in that include: A detection module, used to perform target detection on the first image to obtain the center point coordinate information of the target object in the first image and the size information of the target object; A first determination module, used to determine the coordinates of the truncation point according to the center point coordinate information of the target object and the size information of the target object; A second determination module, configured to determine a truncation area in the first image according to the truncation point coordinates and the truncation direction; A processing module, configured to perform truncation processing on the truncation area in the first image to obtain a second image; The first determining module is further used for: Determine a truncation point coordinate value range according to a preset truncation range coefficient, the center point coordinate information of the target object, and the size information of the target object; Select a value within the truncation point coordinate value range to obtain the truncation point coordinate; The truncation point coordinates and the truncation direction are determined according to the following formula: Wherein, η is the preset cutoff range coefficient; U is the probability distribution followed by the sampling, specifically uniform distribution; represents n different optional truncation directions for truncating the first image; are the coordinates of the cutoff point; is the truncation direction; (c x ,c y ) is the coordinate of the center point of the target object; l is the length of the target object; w is the width of the target object; The value range is [c x -η·l,c x +η·l]; The value range is [c y -η·w,c y +η·w]; The second determining module is further used for: Determine a first pixel point according to the coordinates of any pixel point in the first image, the coordinates of the truncation point, and the truncation direction; All the first pixel points constitute an initial truncation area; Determining a truncation region in the first image according to the initial truncation region; The second determining module is further used for: Determine a truncation area in the first image according to the center point coordinate information of the target object, the size information of the target object and the initial truncation area; Wherein, when the truncation direction is upward, it is determined that the ordinate in the initial truncation area is located at With w / 2+c y The pixel points between are the pixel points in the truncated area, so as to obtain the truncated area; when the truncation direction is downward, it is determined that the ordinate in the initial truncation area is located at With w / 2-c y The pixel points between are the pixel points in the truncated area, so as to obtain the truncated area; when the truncation direction is to the left, it is determined that the horizontal coordinate in the initial truncation area is located at With c x -l / 2 are the pixel points in the truncated area, so as to obtain the truncated area; when the truncation direction is rightward, it is determined that the horizontal coordinate in the initial truncation area is located at With c x The pixel points between +l / 2 are the pixel points in the truncation area, thereby obtaining the truncation area.

7. A neural network training device for object posture estimation, It is characterized in that include: A training module is used to train a neural network according to a preset training set, wherein the sample images in the training set are obtained after truncation processing using the image processing method according to any one of claims 1 to 4.

8. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

9. A computer-readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images