Image cropping method, image acquisition device and storage medium

Abstract

The application discloses an image cropping method, an image acquisition device and a storage medium, relates to the technical field of image processing, and discloses an image cropping method, which comprises the following steps: determining a face existence detection result of a current image photographed by an image acquisition device based on a face detection head of a CNN convolutional neural network; determining predicted cropping point coordinates based on a cropping point positioning head parallel to the face detection head in the CNN convolutional neural network; and if the face existence detection result is that there is a face, cropping the current image based on the predicted cropping point coordinates. Through the architecture of the parallel work of the double-task head of the CNN convolutional neural network after image shooting, face existence detection and positioning of cropping coordinates are synchronously completed, the image can be automatically cropped without user operation, the image does not need to be screened one by one, the image processing process is simplified, and the image processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to image cropping methods, image acquisition devices and storage media. Background Technology

[0002] Smart earphones with built-in cameras are gradually gaining market attention as an emerging photography device. Due to the structural characteristics of the earphones and their unique wearing position, the camera is located near the ear, inevitably capturing the wearer's face in the photo. This poses significant privacy concerns and user experience issues for users.

[0003] In current headphone image processing methods, images are typically manually cropped after capture or post-processed using general image editing software. However, this approach requires image-by-image screening and manipulation, making the process cumbersome. Therefore, capturing images from headphones suffers from low image processing efficiency.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide an image cropping method, an image acquisition device, and a storage medium, which aims to solve the technical problem of cumbersome processing procedures and low processing efficiency when manually cropping or post-processing images after they are captured.

[0006] To achieve the above objectives, this application proposes an image cropping method, which includes: A face detection head based on a CNN convolutional neural network determines the face presence detection result in the current image captured by the image acquisition device; The predicted cropping point coordinates are determined based on the cropping point localization head that runs parallel to the face detection head in the CNN convolutional neural network. If the face presence detection result indicates the presence of a face, the current image is cropped based on the predicted cropping point coordinates.

[0007] In one embodiment, the step of determining the predicted cropping point coordinates based on the cropping point localization head, which is parallel to the face detection head in the CNN convolutional neural network, includes: Based on the image features obtained after feature extraction from the current image in the CNN convolutional neural network, the target to be cropped is determined; Pixel-level regression is performed on the horizontal pixel position of the target to be cropped based on a two-dimensional coordinate system to obtain the coordinate value of the target on the horizontal axis, and the coordinate value is determined to be the coordinate of the predicted cropping point.

[0008] In one embodiment, after the step of determining the target to be cropped based on the image features obtained by feature extraction from the CNN convolutional neural network on the current image, the image cropping method further includes: Obtain the wearing position or placement position of the image acquisition device; The initial orientation of the two-dimensional coordinate system is determined based on the wearing position or the placement position, and the two-dimensional coordinate system is constructed in the current image based on the initial orientation.

[0009] In one embodiment, before the step of the face detection head based on a CNN convolutional neural network determining the face presence detection result of the current image captured by the image acquisition device, the image cropping method further includes: Convert the storage format of the model weights of the convolutional and fully connected layers of the CNN convolutional neural network to the first data precision format; The bias values ​​of the model weights of the convolutional layer and the fully connected layer are converted into a second data precision format, the storage bit width of the second data precision format being greater than that of the first data precision format.

[0010] In one embodiment, before the step of the face detection head based on a CNN convolutional neural network determining the face presence detection result of the current image captured by the image acquisition device, the image cropping method further includes: Acquire the current image captured by the image acquisition device, and compress the current image based on a preset resolution; The compressed current image is input into the CNN convolutional neural network, and the image features of the compressed current image are extracted based on the feature sharing layer of the CNN convolutional neural network.

[0011] In one embodiment, after the step of determining the predicted cropping point coordinates based on the cropping point localization head parallel to the face detection head in the CNN convolutional neural network, the image cropping method further includes: If the face presence detection result is that no face exists, and the predicted cropping point coordinates are not empty, determine whether the predicted cropping point coordinates are within the effective pixel range of the current image; If so, based on the face detection head, the predicted cropping point coordinates are located in the region, and a secondary detection result is obtained; The detection weights of the face detection head and / or the cropping point positioning head are updated based on the secondary detection results.

[0012] In one embodiment, before the step of the face detection head based on a CNN convolutional neural network determining the face presence detection result of the current image captured by the image acquisition device, the image cropping method further includes: Obtain the image features of the current image after feature extraction by the CNN convolutional neural network; Based on the image features, feature scores are determined for the facial features, image noise, and facial pose offset of the current image, respectively. Based on the feature weights calculated from the feature scores, the detection weights of the face detection head and the cropping point positioning head are determined.

[0013] In one embodiment, after the step of determining the detection weight of the face detection head and the detection weight of the cropping point positioning head based on the feature weights calculated from the feature scores, the image cropping method further includes: The human voice detection result is determined based on the audio signal acquired by the image acquisition device, and the device status of the image acquisition device is determined based on the inertial sensing information of the image acquisition device. If the voice detection result indicates the presence of a human voice, and / or the device is in motion, the detection weight of the face detection head is increased based on a preset weight.

[0014] In addition, to achieve the above objectives, this application also proposes an image acquisition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image cropping method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the image cropping method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: By setting up parallel face detection and cropping point localization heads within a CNN convolutional neural network, the two heads can share features extracted by the feature extraction layer and simultaneously complete both face detection and cropping point localization tasks. Finally, upon detecting a face, the image is cropped based on the predicted cropping point coordinates. After image capture, this dual-task-head parallel architecture synchronously completes face presence determination and cropping point localization within a single CNN inference step, eliminating the need for additional independent inference steps. This reduces the overall computational burden of image processing and lowers device power consumption. Furthermore, the cropped image is automatically obtained after user capture without any additional operation, eliminating reliance on manual intervention. Users no longer need to sift through images, manually adjust cropping ranges, or switch to professional editing software, significantly simplifying the image processing workflow and improving efficiency. Moreover, image cropping is performed locally on the device after acquisition, avoiding privacy leaks caused by uploading images to the cloud for processing. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the image cropping method of this application; Figure 2 A schematic diagram of the image captured by the right earpiece in the image cropping method of this application; Figure 3 This is a schematic diagram of the image cropping method of this application; Figure 4 This is a schematic diagram of the CNN convolutional neural network processing used in the image cropping method of this application; Figure 5 A simplified flowchart illustrating the image cropping method provided in the third embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the image cropping method in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] The main solution of this application embodiment is: a face detection head based on a CNN convolutional neural network determines the face presence detection result of the current image captured by the image acquisition device; The predicted cropping point coordinates are determined based on the cropping point localization head that runs parallel to the face detection head in the CNN convolutional neural network. If the face presence detection result indicates the presence of a face, the current image is cropped based on the predicted cropping point coordinates.

[0023] In this embodiment, for ease of description, the following description uses an image acquisition device as the execution subject.

[0024] In current headphone image processing methods, images are typically manually cropped after capture or post-processed using general image editing software. However, this approach requires image-by-image screening and manipulation, making the process cumbersome. Therefore, capturing images from headphones suffers from low image processing efficiency.

[0025] Therefore, this application provides a solution by setting up a face detection head and a cropping point localization head in parallel processing within a CNN convolutional neural network. The two detection heads can share features extracted by the feature extraction layer and simultaneously complete both face detection and cropping point localization tasks. Finally, when a face is detected, the current image is cropped based on the predicted cropping point coordinates. Based on the architecture of parallel operation of the dual task heads after image capture, face existence determination and cropping point localization are completed synchronously during a single CNN inference process, eliminating the need for additional independent inference steps. This reduces the overall computational burden of image processing and lowers device power consumption. Furthermore, the cropped image is automatically obtained after user capture without any additional operation, eliminating reliance on manual operation. Users no longer need to sift through images one by one, manually adjust the cropping range, or switch to professional editing software, significantly simplifying the image processing workflow and improving image processing efficiency. Simultaneously, image cropping is performed locally on the device after image acquisition, avoiding privacy leaks caused by uploading images to the cloud for processing.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or image acquisition device capable of performing the above functions. The image acquisition device can be smart glasses, an action camera, a law enforcement recorder, a vehicle-mounted camera, or headphones with a camera.

[0027] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0028] This application provides an image cropping method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image cropping method of this application, wherein an earphone with a camera device is used as an example to illustrate this embodiment and the following embodiments.

[0029] In this embodiment, the image cropping method includes steps S10 to S30: Step S10: The face detection head based on CNN convolutional neural network determines the face presence detection result of the current image captured by the image acquisition device.

[0030] CNN (Convolutional Neural Network) is a deep learning network that includes convolutional layers, pooling layers, and fully connected layers. It performs image analysis through local feature extraction and hierarchical abstraction. The face detection head is a fully connected layer in the CNN specifically designed to output a binary classification result indicating the presence of a face.

[0031] The CNN convolutional network in this embodiment is a dual-detector convolutional neural network model, which includes a feature extraction backbone network composed of multiple layers of CNN convolutional neural networks. Through the stacking of convolutional layers, pooling layers and activation functions, it gradually extracts the abstract feature representation of the image.

[0032] As an optional implementation, the headphones capture images of the current scene in real time via a camera, perform noise reduction preprocessing on the images locally, and then input the preprocessed images into the feature extraction backbone of a CNN (Convolutional Neural Network). The feature extraction backbone extracts multi-scale features. After multi-scale feature extraction, a face detection head performs classification operations on the feature maps output by the feature extraction backbone. Through a fully connected layer and a sigmoid activation function, the high-dimensional features are mapped to confidence values ​​in the [0,1] interval to obtain the face presence detection result. A face is considered present when the confidence level is greater than or equal to a preset threshold. When the face detection head combines a fully connected layer (FC) with a Softmax activation function to output a binary classification result for face presence, the cross-entropy loss function is calculated as follows: .

[0033] For the loss value, the parameters This represents the true label of the i-th sample.

[0034] As an alternative implementation, after the headphones continuously acquire images, the images can be normalized. Then, image features are extracted through convolutional and pooling layers of the feature extraction backbone network. After feature extraction, anchor boxes of three sizes (16×16, 32×32, and 64×64) are generated by the face detection head to cover face targets of different sizes, avoiding the omission of small or distant faces. The anchor boxes are then matched with the feature maps output by the feature extraction backbone network, and the face matching probability of each anchor box is calculated using a Softmax classifier, thus obtaining the face presence detection result.

[0035] Step S20: Determine the predicted cropping point coordinates based on the cropping point localization head that runs parallel to the face detection head in the CNN convolutional neural network.

[0036] In this embodiment, the cropping point localization head and the face detection head share a parallel functional branch of the CNN feature extraction backbone network to regress the key coordinates of face cropping. Therefore, as a parallel branch, steps S10 and S20 are executed simultaneously. Both share the multi-scale features extracted by the CNN feature extraction backbone network. In this way, by sharing the feature extraction network, the two tasks of face detection and cropping point localization are completed. Compared with training two models separately, the number of parameters is reduced by about 50%, which significantly improves the inference speed of the model.

[0037] The predicted cropping point coordinates are the coordinates output by this branch. These can be X-axis pixel coordinates, Y-axis pixel coordinates, a two-dimensional coordinate pair consisting of X-axis and Y-axis pixel coordinates, the diagonal coordinates of the rectangular cropping box, coordinates of multiple key points in the face's core region, or relative proportional coordinates based on the face bounding box. It should be noted that there can be one or more predicted cropping point coordinates. If the image does not contain the target to be cropped, such as features like cheeks or noses, the predicted cropping point coordinates are set to empty, meaning no subsequent cropping processing is performed.

[0038] In this embodiment, considering the special scenario of taking photos with the earphones, based on the earphones' own structure and the user's wearing habits, the captured image may be partially obscured by a person's face. For example, the image captured by the right earphone may be partially obscured by a person's face. Figure 2 The image shown is currently obscured by the user's right cheek on the left side. When cropping the image, please refer to... Figure 3 This method only requires cropping based on pixel values ​​along the X-axis, without cropping along the Y-axis, and the cropping position is the largest side of the face in the image. Therefore, only the cropping point of the largest side needs to be marked as the target point. During the cropping process, only one coordinate value, the X-coordinate, needs to be output for subsequent cropping processing, which greatly simplifies the complexity of the regression task. In actual headphone shooting scenarios, it can avoid redundant calculations of general solutions and ensure that the maximum effective image content is retained after cropping.

[0039] Therefore, as an optional implementation, the predicted cropping point coordinates are the X-axis pixel coordinates. When determining the predicted cropping point coordinates, the cropping point localization head can determine the target to be cropped based on the image features extracted from the current image in the CNN convolutional neural network. Then, it performs pixel-level regression on the horizontal pixel position of the target to be cropped based on a two-dimensional coordinate system to obtain the coordinate value of the target on the horizontal axis, and determines this coordinate value as the predicted cropping point coordinate. The starting point of the two-dimensional coordinate system can be based on the wearing position of the headphones. For example, when the headphones are worn in the left ear, the right side of the image is easily obscured by the side of the face; in this case, the starting point of the two-dimensional coordinate system is set from the right side of the image. Therefore, after determining the cropping coordinates, the wearing position of the image acquisition device can also be obtained. Then, the initial orientation of the two-dimensional coordinate system is determined based on the wearing position. Finally, a two-dimensional coordinate system is constructed in the current image based on the initial orientation, providing a coordinate basis for the generation of the predicted cropping point.

[0040] Optionally, when the image acquisition device is not a headset, the placement position of the device can be obtained, and then the initial orientation of the two-dimensional coordinate system can be determined based on the placement position. Finally, the two-dimensional coordinate system can be constructed in the current image based on the initial orientation. For example, when the image acquisition device is an action camera placed on the shoulder, if it is placed on the right shoulder, the direction on the left side, which is more likely to capture the side profile due to the user's movement, can be used as the initial orientation, and the two-dimensional coordinate system can be constructed based on this orientation.

[0041] Furthermore, when the cropping point positioning head outputs the X-coordinate position of the cropping point based on the regression head, in order to quantify the prediction accuracy, the network parameters are optimized through mean squared error loss, thereby achieving accurate positioning of the x-axis coordinate. The formula for calculating the mean squared error (MSE) loss function is as follows: , This is the loss value. This represents the true label of the cropping point x-coordinate in the i-th sample.

[0042] For example, based on image features extracted by the CNN backbone network, the cropping point localization head focuses on the core region of the face through a spatial attention mechanism, and identifies the features of key facial organs in the feature map through a pre-trained organ point detection sub-network. At the same time, it constructs a two-dimensional coordinate system with the upper left corner of the image as the origin, and clarifies the pixel value range of the horizontal axis. Then, the feature map after locking the target to be cropped is compressed into a one-dimensional feature vector through a global average pooling layer, and input into two fully connected layers for feature transformation. Finally, the output of the fully connected layer is normalized to the [0, 1] interval through the Sigmoid activation function, and then multiplied by the image width pixel value to obtain the pixel-level coordinate value of the target to be cropped on the horizontal axis. This coordinate value is the predicted cropping point coordinate.

[0043] Optionally, the cropping point positioning head can also detect key facial organ points, including eyes, nose, and jaw angle, through convolutional layers. Then, K-means clustering is performed on the detected key organ points to filter out invalid points that are occluded or falsely detected, retaining 3 core reference points. Finally, the X-axis coordinates of the cropping point are calculated based on the geometric proportion constraints of the face.

[0044] Understandably, during training, the two parallel detection heads share the backbone of the CNN. The joint loss function allows the feature extraction layer to simultaneously learn general features useful for both face detection and cropping point localization. Therefore, the joint loss function enables the two tasks to collaborate and share optimizations during training, thereby improving overall performance. The joint loss function is the model's total loss function, which is a weighted sum of the losses from both tasks. The calculation formula is as follows: , This is the total loss value. The corresponding weights.

[0045] This embodiment is based on a CNN parallel architecture, reusing features from the same backbone network, eliminating the need for independent feature extraction. This reduces computational redundancy and ensures accurate correlation between cropping point coordinates and the facial target. Simultaneously, a cropping point localization head determines the optimal cropping point location, ensuring that the cropped image removes facial information while retaining maximum effective capacity. Step S30: If the face presence detection result indicates the presence of a face, crop the current image based on the predicted cropping point coordinates.

[0046] In this embodiment, for the image captured by the earphone, when the face presence detection result indicates the presence of a face, vertical cropping can be performed based on the horizontal axis pixel coordinates, that is, in Figure 3 The cut lines shown are used to crop the current image. It is understandable that after image cropping is complete, Figure 3 The face region shown is removed as a redundant region, and the remaining part of the image is the cropped effective region.

[0047] As an optional implementation, the predicted cropping point is usually single, meaning that a face occlusion occurs on one side of the image. Therefore, vertical cropping can be directly performed based on the X-axis coordinate of the predicted cropping point. In the scenario of shooting with headphones, vertical cropping is performed based on the optimal cropping point position, i.e., the predicted cropping point, to ensure that the cropped image removes facial information while retaining the largest effective area, avoiding under-cropping or insufficient cropping. As another alternative implementation, in multi-person interaction scenarios such as users wearing headphones to participate in multi-person video conferences, record multi-person videos, or go out to queue, the headphones may capture multiple faces. That is, when there are multiple predicted cropping points, cropping processing needs to be performed sequentially based on the predicted cropping points.

[0048] Optionally, if the face presence detection result indicates that no face exists, no further processing is required.

[0049] For example, the process of image processing using a dual-detector-head CNN convolutional neural network is as follows: Figure 4 As shown, after the image is input into the CNN convolutional neural network, features are extracted through multiple rounds of convolutional layers (CONV). Then, a fully connected layer (FC) completes a dual-task judgment: predicting the cropping point and determining whether a face exists in the image. Both are performed simultaneously. Finally, if a face exists, the input image is cropped based on the predicted cropping point, and the cropped image is output.

[0050] This embodiment provides an image cropping method. By setting up a cropping point localization head parallel to the face detection head in a CNN convolutional neural network, the face detection head determines the presence of faces in the current image captured by the image acquisition device. Simultaneously, the parallel cropping point localization head determines the predicted cropping point coordinates. Finally, based on these predicted cropping point coordinates, the current image is automatically cropped. The entire process requires no manual user intervention or post-processing software, eliminating the tedious steps of image-by-image filtering and manual selection of cropping areas, thus improving detection efficiency. Furthermore, it allows users to directly obtain a focused image of the target face after capturing an image using headphones, perfectly meeting the convenience requirements of headphone use.

[0051] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment can be referred to the above description, and will not be repeated hereafter. Furthermore, before step S10, the image cropping method further includes steps S01~S02: Step S01: Convert the storage format of the model weights of the convolutional and fully connected layers of the CNN convolutional neural network into the first data precision format.

[0052] Step S02: Convert the bias values ​​of the model weights of the convolutional layer and the fully connected layer into a second data precision format.

[0053] The storage bit width of the second data precision format is greater than that of the first data precision format.

[0054] It should be noted that data storage formats typically include common formats such as FP64, FP32, FP16, INT16, and INT8. In CNN convolutional neural networks, the data storage format for the model weights of convolutional layers and fully connected layers is usually FP32. However, in order to compress the model size and accelerate the model inference speed, this embodiment needs to perform segmented quantization on the model.

[0055] Therefore, as an optional implementation method before capturing images, the first data precision format is INT8, and the second data precision format is FP32. The data storage format of the model weights of the convolutional and fully connected layers can be converted to INT8, while the bias values ​​of the model weights of the convolutional and fully connected layers can be converted to FP32. It should be noted that if the current format is the converted format, no conversion processing is required. The specific conversion process is existing technology and will not be elaborated upon in this application.

[0056] This embodiment uses segmented quantization to quantize the model weights from FP32 to INT8, while keeping the model bias at FP32. This compresses the model size, reduces accuracy loss, accelerates model inference speed, and improves the battery life, real-time interactive response efficiency, and intelligent processing stability of lightweight products such as headphones with cameras. It also reduces the product's dependence on high-performance chips and large-capacity batteries. While maintaining the core characteristics of compact portability and low power consumption of headphones, it ensures the accurate execution of tasks such as face detection and cropping point localization. This eliminates user concerns about insufficient battery life or hardware overload, further expanding the application range and practical value of lightweight video headphones.

[0057] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment can be referred to the above description, and will not be repeated hereafter. Furthermore, before step S10, the image cropping method further includes steps S03~S04: Step S03: Acquire the current image captured by the image acquisition device and compress the current image based on a preset resolution; Step S04: Input the compressed current image into the CNN convolutional neural network, and extract the image features of the compressed current image based on the feature sharing layer of the CNN convolutional neural network.

[0058] It should be noted that, in order to fully adapt to the limited computing power, storage and power consumption constraints of lightweight products such as headphones, the model architecture of CNN convolutional neural networks needs to be set to a lightweight architecture. Small-sized image input can reduce hardware resource usage, avoid redundant calculations, reduce device power consumption and extend the battery life of headphones when processing lightweight models.

[0059] Therefore, in this embodiment, after acquiring the current image captured by the image acquisition device, before inputting the current image into the CNN for processing, it is necessary to compress the current image based on a preset resolution before inputting the compressed image into the CNN model. This allows the feature sharing layer of the CNN convolutional neural network to extract the image features of the compressed current image. For example, if the resolution of the image captured by the headphones is 1k, when inputting it into the lightweight CNN model, the model resolution needs to be adjusted to a preset image size of 320×240 pixels.

[0060] It should be noted that image cropping involves cropping the original high-definition image.

[0061] For example, to help understand the implementation flow of the image cropping method obtained by combining this embodiment with the first embodiment described above, please refer to... Figure 5 , Figure 5 A simplified flowchart of an image cropping method is provided. Specifically: In the image input stage, a camera image captured by a headphone camera is received. Then, in the image preprocessing stage, resizing (scaling the image to 320×240 resolution), normalization, and grayscale conversion are performed. Next, in the model algorithm inference stage, the preprocessed image is input into a dual-detector network, simultaneously obtaining the face presence detection value Fdetection and the cropping point location Pboundary (i.e., detection of face presence and cropping location). A cropping decision is then made: if Fdetection=0 (no face), the original image is retained without cropping; if Fdetection=1 (face), cropping is performed based on the Pboundary (cropping location). During cropping, the original high-resolution image is directly cropped to remove the face portion. Finally, the cropped image is output for users to view and share.

[0062] Based on any of the above embodiments, in the fourth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. On this basis, after step S20, in an extreme case, if the face existence detection result is that there is no face, but the predicted cropping point coordinates are not empty, it indicates that there is a contradiction in the parallel head output of the CNN neural network.

[0063] It's important to note that computing power / power consumption is a core constraint for lightweight devices like headphones with cameras. When a CNN convolutional neural network simultaneously executes a face detection head and a cropping / localization head, the face detection head performs a binary classification task, only needing to distinguish between faces and non-faces, relying on the discriminative power of features to make the decision, allowing for a lightweight model structure. However, the cropping / localization head performs a pixel-level regression task, requiring accurate prediction of horizontal coordinates, demanding higher precision and continuity of features, and requiring the model to handle more complex mapping and calibration operations. Therefore, for two detection heads operating simultaneously, to avoid wasting or insufficient computing power while ensuring high accuracy in face detection and precise cropping points, the image acquisition device can allocate different computing power to the detection heads based on task complexity. For example, the face detection head typically allocates 30%-40% of the computing power / power, while the cropping / localization head allocates 60%-70%. Therefore, when the outputs of the two detection heads contradict each other, it's generally considered that the computing power allocation is insufficient, leading to inadequate feature extraction or excessive computational time without sufficient support, resulting in contradictory results from one of the detection heads.

[0064] Building upon this, when inconsistencies arise in the parallel head outputs of a CNN neural network, rapid verification is required to identify the erroneous end. During verification, the rationality of the cropping point can be checked first, verifying whether the predicted cropping point coordinates are within the valid pixel range of the current image, or whether a valid cropping region can be formed. If the coordinates are outside the range or cannot form a valid region, the cropping point localization head is directly deemed to have detected an error. When the cropping point localization head detects an error, its detection weight can be increased, such as from 60% to 70%. The adjusted parameters can be set manually or adjusted automatically by the model, such as through auto-increment / auto-checking based on preset parameters, or adjustment based on preset rules.

[0065] Optionally, when the coordinates are within the specified range, secondary verification is required based on the face detection head. The face detection head identifies and predicts the area where the cropping point coordinates are located, obtaining a secondary detection result. The detection weights of the face detection head and / or the cropping point localization head are then updated based on this result. For example, if low-dimensional features such as facial contours or skin texture are detected in the secondary detection result, it is determined that the detection head missed the detection, and the detection weight of the face detection head is increased while the detection weight of the cropping point localization head is decreased. The implementation method is similar in other cases, and will not be elaborated upon in this application.

[0066] This embodiment performs secondary verification when the results of the detection heads processed simultaneously are incorrect, and dynamically adjusts the detection weights based on the results of the secondary verification. This ensures the accuracy of the detection while avoiding waste or insufficiency of the equipment's computing power.

[0067] Based on any of the above embodiments, in the fifth embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. In addition, before step S10, steps S05~S07 are also included: Step S05: Obtain the image features of the current image after feature extraction by the CNN convolutional neural network.

[0068] In this embodiment, to improve detection accuracy or precision, the computational weights of the parallel heads can be adjusted based on the scene information of the image. For example, when facial features are clear in the image, the weight of the cropping point positioning head is increased to improve coordinate accuracy; when the image is noisy and facial features are weak, the weight of the face detection head is increased to ensure the accuracy of presence detection and avoid incorrect or missed cropping.

[0069] Therefore, please continue to refer to Figure 4 Before the fully connected layer (FC) processing, a dynamic weight adjustment module (not shown in the figure) is added to the face detection head and the cropping point localization head to adjust the inference weights of the two parallel heads in real time according to the scene features of the current image, such as light intensity, face pose, and image blur.

[0070] Step S06: Based on image features, determine the feature scores corresponding to the face features, image noise, and face pose offset of the current image.

[0071] The feature score is a normalized score value. In this embodiment, quantifiable features include facial features, image noise, and facial pose shift. A feature score for facial features (0-1) can be obtained by performing global average pooling on the facial anchor box region in the feature map and combining it with a binarized mask; high-frequency components can be extracted from the feature map, and the high-frequency variance can be calculated and normalized to obtain the feature score for image noise; simultaneously, key points for both eyes / nose tip can be extracted from the feature map, and pitch / yaw angles can be calculated and normalized to obtain the feature score for facial pose shift.

[0072] Step S07: Determine the detection weight of the face detection head and the detection weight of the cropping point positioning head based on the feature weights calculated by the feature scores.

[0073] In this embodiment, based on the calculated feature scores, the face detection head weight W_d and the cropping point localization head weight W_c are calculated using a nonlinear mapping function.

[0074] Wherein, W_d=α×(1-S)+β×N+γ×A; W_c = 1 - W_d; α, β, and γ are learnable coefficients, which are optimized iteratively through the loss function. S is the score corresponding to the face features, N is the score corresponding to the image noise, and A is the score corresponding to the face pose shift.

[0075] It should be noted that the detection weight is usually between [0.2, 0.8] to avoid the function failing due to the weight of a single detection head being too low.

[0076] For example, with initial values ​​α=0.4, β=0.3, and γ=0.3, when the image is clear, S=0.95, N=0.1, and A=0.1. At this time, W_d=0.4×0.05+0.3×0.1+0.3×0.1=0.08. After the weight range constraint, the actual W_d=0.2 and W_c=0.8, that is, the weight of the cropping point positioning head is increased to 0.8 to improve the cropping accuracy.

[0077] This embodiment dynamically adjusts the detection weights of two detection heads performing tasks simultaneously, thereby balancing detection reliability and cropping accuracy in complex scenarios without adding extra inference branches, maintaining lightweight characteristics. Compared to fixed weights, if the weights are biased towards the face detection head, the cropping point coordinate error will be large in clear scenes, missing key features; if the weights are biased towards the cropping point localization head, the face detection false positive or false negative rate will be high in complex scenes.

[0078] Furthermore, after determining the detection weights, the effectiveness of weight adjustment can be improved through collaborative analysis of the device's status and the acquired audio. Specifically, the human voice detection result can be determined based on the audio signal acquired by the image acquisition device, and the device status of the image acquisition device can be determined based on the inertial sensing information of the image acquisition device. When the human voice detection result indicates the presence of human voice, and / or the device status indicates motion, the detection weight of the face detection head needs to be increased based on preset weights. This reduces the occurrence of missed or false face detections in noisy or moving scenes.

[0079] This application provides an image acquisition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image cropping method described in the first embodiment above.

[0080] The following is for reference. Figure 6The diagram illustrates a structural schematic of an image acquisition device suitable for implementing embodiments of this application. The image acquisition device in the embodiments of this application may include, but is not limited to, mobile terminals such as headphones equipped with cameras, mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The image acquisition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0081] like Figure 6 As shown, the image acquisition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the image acquisition device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the image acquisition device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows image acquisition devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0082] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0083] The image acquisition device provided in this application, employing the image cropping method described in the above embodiments, can solve the technical problem of cumbersome processing procedures and low processing efficiency caused by manual cropping or post-processing after image capture. Compared with the prior art, the beneficial effects of the image acquisition device provided in this application are the same as those of the image cropping method provided in the above embodiments, and other technical features of this image acquisition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0084] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0086] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the image cropping method described in the above embodiments.

[0087] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM, or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0088] The aforementioned computer-readable storage medium may be included in the image acquisition device; or it may exist independently and not assembled into the image acquisition device.

[0089] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the image acquisition device, cause the image acquisition device to: A face detection head based on a CNN convolutional neural network determines the face presence detection result in the current image captured by the image acquisition device; The predicted cropping point coordinates are determined based on the cropping point localization head that runs parallel to the face detection head in the CNN convolutional neural network. If the face presence detection result indicates the presence of a face, the current image is cropped based on the predicted cropping point coordinates.

[0090] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0093] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image cropping method. This solves the technical problem of cumbersome processing procedures and low processing efficiency during manual cropping or post-processing of images after capture. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image cropping method provided in the above embodiments, and will not be repeated here.

[0094] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image cropping method, characterized by, The image cropping method includes: A face detection head based on a CNN convolutional neural network determines the face presence detection result in the current image captured by the image acquisition device; Determining predicted cropping point coordinates based on a cropping point localization head parallel to the face detection head in the CNN convolutional neural network includes: determining the target to be cropped based on image features obtained after feature extraction from the current image in the CNN convolutional neural network; obtaining the wearing position or placement position of the image acquisition device; determining the initial orientation of the two-dimensional coordinate system based on the wearing position or placement position, and constructing the two-dimensional coordinate system in the current image based on the initial orientation; performing pixel-level regression on the horizontal pixel position of the target to be cropped based on the two-dimensional coordinate system to obtain the coordinate value of the target to be cropped on the horizontal axis, and determining the coordinate value as the predicted cropping point coordinates; If the face presence detection result indicates the presence of a face, the current image is cropped based on the predicted cropping point coordinates.

2. The image cropping method as described in claim 1, characterized in that, Before the step of determining the face presence detection result of the current image captured by the image acquisition device using the CNN-based convolutional neural network face detection head, the image cropping method further includes: Convert the storage format of the model weights of the convolutional and fully connected layers of the CNN convolutional neural network to the first data precision format; The bias values ​​of the model weights of the convolutional layer and the fully connected layer are converted into a second data precision format, the storage bit width of the second data precision format being greater than that of the first data precision format.

3. The image cropping method according to any one of claims 1 to 2, characterized in that, Before the step of determining the face presence detection result of the current image captured by the image acquisition device using the CNN-based convolutional neural network face detection head, the image cropping method further includes: Acquire the current image captured by the image acquisition device, and compress the current image based on a preset resolution; The compressed current image is input into the CNN convolutional neural network, and the image features of the compressed current image are extracted based on the feature sharing layer of the CNN convolutional neural network.

4. The image cropping method as described in claim 1, characterized in that, After the step of determining the predicted cropping point coordinates based on the cropping point localization head parallel to the face detection head in the CNN convolutional neural network, the image cropping method further includes: If the face presence detection result is that no face exists, and the predicted cropping point coordinates are not empty, determine whether the predicted cropping point coordinates are within the effective pixel range of the current image; If so, based on the face detection head, the predicted cropping point coordinates are located in the region, and a secondary detection result is obtained; The detection weights of the face detection head and / or the cropping point positioning head are updated based on the secondary detection results.

5. The image cropping method as described in claim 1, characterized in that, Before the step of determining the face presence detection result of the current image captured by the image acquisition device using the CNN-based convolutional neural network face detection head, the image cropping method further includes: Obtain the image features of the current image after feature extraction by the CNN convolutional neural network; Based on the image features, feature scores are determined for the facial features, image noise, and facial pose offset of the current image, respectively. Based on the feature weights calculated from the feature scores, the detection weights of the face detection head and the cropping point positioning head are determined.

6. The image cropping method as described in claim 5, characterized in that, After the steps of determining the detection weight of the face detection head and the detection weight of the cropping point positioning head based on the feature weights calculated according to the feature scores, the image cropping method further includes: The human voice detection result is determined based on the audio signal acquired by the image acquisition device, and the device status of the image acquisition device is determined based on the inertial sensing information of the image acquisition device. If the voice detection result indicates the presence of a human voice, and / or the device is in motion, the detection weight of the face detection head is increased based on a preset weight.

7. An image acquisition device, characterized in that, The image acquisition device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image cropping method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image cropping method as described in any one of claims 1 to 6.

Images